Data Lifecycle
Data Self-service Workflow
ROLE: PRODUCT DESIGNER
The design team is tasked with improving the data infrastructure organization's efficiency as Meta moves forward to be more privacy-aware and efficiency-driven. From analytics and qualitative research, we realized that internal employees are creating tons of data artifacts (tables, dashboards, pipelines, metrics, etc.) without maintaining and reusing the current ones. This leads to more and more data artifacts that do not comply with privacy review practices and inefficiencies in power and storage consumption.
Role
As the sole designer on the team, I was tasked with navigating the complexity by creating a holistic experience that can be used by the organization. I worked with four different teams' engineering and product managers and was able to have three of the teams build our solution. The project took less than two months to complete.
Goals
Create a self-service lifecycle management experience for all data asset classes.
Design an easy-to-use flow for users to deprecate tables/dashboards.
Who
Our target users are internal employees who create and maintain data artifacts across teams at Meta. They are usually the following:
Data Scientists: Create dashboards and data artifacts for specific purposes. Familiar with data tooling and SQL.
Data Engineers: Support Data Scientists to make sure the pipelines are reliable and healthy. Familiar with data tooling, coding, and SQL.
Project Managers and others: In some cases when there are no data scientists on the team, project managers, or program managers jump in to create dashboards. Not necessarily familiar with data toolings.
Context
Based on foundational studies conducted by our user researchers on different data asset types, we generalized four different phases that a person goes through for the lifecycle of a data artifact. Let's take a dashboard as an example:
Jon (Data Scientist) is responsible for this year's growth initiative for Instagram Reels.
Create: Jon puts together a dashboard from various metrics, tables with the help of the data engineering team. The dashboard contains their goal metrics, counter metrics, or any alternatives to measure their success.
Use: The Reels team reviews the dashboard weekly. In the case of spikes and drops happening, Jon investigates to figure out the potential reasons and reports findings to the team.
Update: Some tables get broken, Jon and the data engineering team either move the metrics to using other tables or fix the table.
End of life: After the initiative concludes, the team switches focus to other Reels initiatives, and Jon decides to deprecate the dashboard.
Four phases for a data artifact
Problems
We categorized several major problems when managing the lifecycle of the data artifact. Current users usually use a coding tool for deletion.
Communication is painful. People make posts on Workplace as the method for approval.
No incentives for deprecation.
Users would like to roll back if needed.
Tools separation.
The team decided to focus on the end-of-life phase since several tools are already built. Also, the end of life has the most impact on the org for a limited timeframe.
Separate tools for deprecating assets. Top one is manual deletion, bottom one is automatic deletion
The team generalized a workflow to help users with deprecating a data artifact.
Design
I explored four different design directions for End of Life, please view it on the desktop for the optimal experience!
Deprecation Flow: Users get notified when the system detects artifacts to deprecate. Users are able to whitelist certain data artifacts, determine alternatives and message downstream and upstream users automatically to gather feedback.
Deprecation Dashboard: Users get summarized impact on the data tools they are using (directory, dashboards…etc). Users are able to search for deprecating artifacts.
Centralized Deprecation Tool: Users can search for different data assets, see detailed information on metadata, current progress and lineage to see if other people are still depending on their created assets.
View Deprecated Assets: Consistent viewing experience on different data toolings. Users can request to unarchive, view alternative asset (Set by owner) or view historical data.
Impact
The team was able to get three different teams to onboard with our design. We have helped deprecate more than 30% of unused tables, 20% of dashboards. We also launched a MVP to recommend to users what metrics to use instead of creating one.
A separate workstream has been created to tackle the lineage problem.