Metric Investigation
Investigate metrics with one click
Role: Product Designer
Metric Investigation is a tool that allows users to analyze and understand changes in metrics without writing SQL queries. It uses visualization tools to compare different dimensions, which can help explain the cause of changes in a metric, like a spike in daily active users. This can help make the process of investigating metric changes more efficient, as product managers and data scientists can quickly and easily access the information they need without waiting for a report to be generated.
Role
The team includes me as a sole designer, an engineer, a product manager, and a user researcher who mainly consults on the project. Since the tool touches on bunch of other internal tools we also interacted with other internal tooling teams.
The re-design project took less than two months from scoping out the project, iteration and final implementation.
Goals
Employees at Meta use many internal data tools to perform similar tasks. Data scientists spend hours every week on repetitive data analysis, which is time-consuming and tedious. We need an automated solution, rather than writing one-off SQL, so they can focus on more exploratory work. Additionally, the investigation process should be streamlined into a single, easy-to-use workflow that serves as the single source of truth across all internal tools.
Who
Our target users are internal employees who utilize metrics to make decisions. From our foundational study, we categorized all the users into two personas: Data Consumers who look at the analysis and consume reports to make better decisions, and Data Developers who are building dashboards, and pipelines to ensure the data is accurate and reliable.
Workflow
The following diagram illustrates the typical workflow when Data Scientists are investigating into a metric:
The yellow boxes are where our investigation tool fits in.
Process
We started by doing a round of heuristic evaluation on the current investigation tool. We talked to 8 users and categorized major problems for our current tool:
Navigation: Users don’t know where they are and it’s hard to find actionable insights.
Trust: Data scientists usually have a hypothesis in their mind (It’s usually a country that is the reason for the change). However, our tool hasn’t defaulted to their liking and they can’t customize it.
Acquisition: Hard to set up root cause analysis. Users have to set up auto-run, which is a cumbersome process.
Design and layout: Confusing headers, rarely use sidebars and complex columns.
I worked with the engineering team to brainstorm ideas for how we could make the most impact within the limited time frame. We decided to focus the project on making it easier for users to find actionable insights, as well as updating the design to provide meaningful insights in the context of what the users are doing, regardless of the tool they are using.
Original tool
One quote from our user:
“It's to complicated, I showed the results to our leadership and they were confused“ - Data Analyst
Design
The first design iteration focused on the following:
Match with our internal design system: I did have to customize certain widgets (e.g topline metric card) to show relevant information for our users.
Progressive disclosure: by looking at the data, we know users don't use comments and left sections much. The design also shows a few important columns so that users can learn as they customize their layout.
Summary section: By utilizing our back-end, we introduce a summary section where we summarize what happens to the metric and what are the potential reasons.
Navigation: The design also introduces a tab structure to minimize the pain of long vertical scrolling area.
Iteration
Throughout the usability testing of the initial design, we realized that users wanted to see the summary section on other surfaces and tools, as their analysis often takes place in notebooks, dashboards, or presentations.
Results
We launched the new design and got a promising impact within a month
178% increase in metrics monitored
92% service engage users growth
50% Impression growth
UI used to be the top two complaints from previous surveys, now it’s dropped to fifth
Reflections
This project was a great learning lesson for me as well!
Success metrics aren't always reflecting real usage: ever since we launched our new design, the number of active users of our tool has actually decreased. The reason for this is that people are spending less time on our tool since they find the summary section sufficient. This means that we should have revisited how we define success.
Onboard everyone as early as possible, even before the planning starts.