Data Literacy
This post is intended for those who are in the roles of defining, analyzing, and presenting metrics; and for those who have been or are in the ‘data puking’ phase, which is very easy to slip into given the abundance of data we have access to today.
I hope this post can provide a framework for you to thread your thoughts, to understand which metrics matter, to whom, and when.
Goal Metrics
Goal metrics are proxies of what the company ultimately cares about. They are usually high-level north-star metrics set by leaderships to align and guide the entire company, to make sure everyone is working towards the same strategic direction.
Goal metrics should be simple and easily understood. They should also be stable and usually take a long time to materialize, with each initiative only has a small impact on it.
- Examples: # of Customers, Share of $ market, Customer satisfaction (NPS)
- Audience: Executives
- Frequency: monthly/quarterly/yearly
Key Driver Metrics
With goal metrics established, each team then can decide on what it takes for them to influence the goal metrics to come up with the key driver metrics.
Key driver metrics are the drivers of success. They tend to be shorter-term and more sensitive than goal metrics. A good key driver metric should be aligned with the goal metrics, actionable, and not easily gameable (e.g. measure qualified leads vs. leads, which can be easily gamed by spending more $$ but won’t necessarily turn into customers, the goal metric).
Key driver metrics are indicators of if the team is moving in the right direction to drive success.
- Examples: # of qualified leads, # of active free-trial users, customer retention rate
- Audience: Team lead, managers, individual contributors
- Frequency: weekly/monthly/quarterly
Guardrail Metrics
Guardrail metrics, like the name suggests, are to make sure things don’t go wrong; meaning while everyone’s attention is on the goal and driver metrics, analysts should monitor guardrail metrics to see if the fundamentals have been impacted along the way.
For example, increasing email campaign frequencies will almost always increase engagement and lead to more revenues, but it might also increase unsubscribe rate. The increase in unsubscribe rate is not reflected in the goal or driver metric but is a possible undesired effect analyst should monitor — a guardrail metric.
Guardrail metrics are usually not proactively reported unless significant changes happen. For experimentations, however, guardrails metrics are included with key driver metrics to form an OEC (Overall evaluation criteria) to effectively evaluate experimentation results.
- Examples: unsubscribe rate, page load time, account deletion rate
- Audience: Analysts and individual contributors unless significant changes happen
- Frequency: daily/weekly/monthly/quarterly
Tactical Metrics (or ‘Vanity Metrics’)
Tactical metrics are metrics that don’t indicate success, and are usually tactic-level performance metrics, hence many people call them vanity metrics.
They are vanity to senior-level stakeholders, however, they are super important for individual contributors to evaluate and improve campaign performances, which will ultimately contribute to improving key driver metrics. Optimization metrics is another name that describes this type of metrics well.
A tactical metric is rarely reported and analyzed alone. They are usually analyzed as a group to explain changes in campaign performance or for root-cause analysis.
- Examples: Impressions, Email Open Rate, Blog post views
- Audience: functional lead, individual contributors
- Frequency: daily/weekly/monthly
If you’ve read this far, you would probably have noticed that the key to knowing what type of metric to measure, analyze, and present is to understand the goal of your stakeholders. Start by having a clear understanding of your stakeholders and their need, then everything will become clearer and easier.
