Introduction to Product Metrics and KPIs - Issue 62
Product and business metrics descriptions, breakdowns, and overviews.
Welcome to a free edition of the Data Analysis Journal, a weekly advice column about data analysis, data science, and business intelligence.
If you’re not a paid subscriber, here’s what you missed this month:
How To Pass A Take-Home Python Assignment - a take-home assignment solution for a senior analyst position. I walk through the steps, the structure of analysis, code, and offer some guidance on how to pass similar take-home challenges.
As The World Churns - what is churn, how to measure and calculate. A synthesis of the most important churn concepts from a product analytical perspective. Churn reporting challenges.
A Selection Of Python Tutorials for Analysts - a list of Python tutorials for data processing, transformations, cleaning, analysis, and visualizations.
For me, September was busy with hiring, preparing Q4 OKRs, learning how to bake a basque cheesecake (which this innocent world will never see again), and catching up on events and updates. In case you missed it, this month Andrew Chen announced his book release - The Cold Start Problem, and Reforge closed applications for Fall 2021 with 14 programs (none of which are solely dedicated to product analytics. With such a powerful group of experts, why not?)
Today’s newsletter is about the foundation of any product analytics - the world of metrics! The most commonly asked and most underrated topic is at the intersection between product, business, and analytics.
I went through hundreds of sources and didn’t find anything that summarises all metrics in one nice structure that is relevant to analytics, so I’ll introduce my own. Here is Olga’s Metrics 101, voted the best and most credible system of metrics classification (all votes were mine):
Metrics Classification
KPIs - Key Performance Indicators are meant to represent a company’s long-term success and performance. KPIs can be financial, customer-focused, and process-focused. They are critical to the company strategy and keep the same definition and logic over time. Often, KPIs are used to measure metrics. Usually, they keep the same definition across whole industries.
Top-level metrics - a measure of strategic direction and performance. Each top-level metric is tailored and adjusted to the specific business model, environment, and strategy. This is when context and nuances really matter. Top-level metrics and KPIs are usually reported monthly and quarterly.
North Star Metric - represents one company goal. From Mixpanel North Star Metric: “To qualify as a “North Star,” a metric must do three things: lead to revenue, reflect customer value, and measure progress” It can be DAU, or LTV, or MRR, or other measurements, and its main purpose is to align teams around one main goal.
Secondary metrics - more granular health indicators and product targets. They measure how successful the product and process are. Secondary metrics are sensitive to any product changes. Therefore, you would measure A/B tests, feature adoptions, or bug impacts against secondary metrics. Due to their sensitivity, they are also usually monitored and reported weekly.
Vanity metrics - impressive but not useful or actionable metrics that don’t lead to growth or revenue and aren’t relevant to anything you can do to improve them. They are often too simple and ignore the context. Examples: the number of social media followers or total registered users. It’s like only working out your arms when you go to the gym and ignoring your core. More about Vanity metrics.
There is also OMTM - One Metric That Matters. This is different from the North Star Metric and meant to be a temporary goal unifying all the teams at the company towards one issue. An example: when your software got hacked and all user accounts got deleted, you would set OMTM as a number of reinstated accounts. For a less dramatic example, when you begin a migration, your OMTM can be the number of successfully migrated accounts. Or when your churn significantly overweights new accounts and renewals, then you have to pause everything and focus on retention.
There is much more to it. Metrics can also be categorized as strategical, operational, or tactical, and as input and output metrics. Some have to be monitored and reported weekly, others monthly. I’ll focus on weekly and monthly metrics in one of my next issues. The more you dig into it, the less you sleep at night the more fun it becomes.
Regarding product domains, here is a list of some (not all) common metrics. Some of them are high-level and common KPIs, others are more sensitive targets for evaluating the product performance.
🌱Growth&Marketing
Unique Visitors, First Visits, Returning Visitors, Bounce Rate, Installs, Signups, Customer Acquisition Cost (CAC), Click Through Rate (CTR), Cost Per Impression (CPI), Cost Per Action (CPA), Time to Value, Visitor to Signup, Signup to Payment, Product or Feature Adoption Rate, Virality, Network Effect Score, Return On Advertising Spend (ROAS), Number of Qualified Leads, Lead Conversion Rate, Average Lead Score, Cost Per Lead (CPL), Unsubscribers
💰Revenue
Monthly Recurring Revenue (MRR), Annual Recurring Revenue(ARR), Net Revenue, Net Revenue Retention, Paid Customers, Activated Trials, Free to Paid, Paid to Free, Revenue Churn, Customer Churn, Monthy/Weekly Customers Completing the First Order, Daily/Monthly Total Purchase Value, Life Time Value (LTV), Average Revenue Per Account, Upsell to Payment, Expansion revenue, Return On Investment (ROI) and more.
😊Engagement
MAU, WAU, DAU, Adjacent Users, Day 0, Day 1+, Day 7+ and Day 28 retention, % 1-year or 2-year retention, Number of Returning Users, Daily/Hourly Number of Actions, Total Watch Time, Total Time Spent, Frequency of Visits, Pages Per Session/Visit, Scroll Depth, Average Session/Visit Duration, Exit Rate / Exit Page, Product Abandonment Rate, others.
💌Customer Success
Customer Satisfaction Score, Net Promoter Score, Customer Health Score, Ticket Resolution Rate, Average Ticket Resolution Time, Average Reply Time, Customer Effort Score, First Response Time, Daily/Monthly Ticket Requests.
⚙️Platform / Engineering
Product Support Cost, R&D Engineering Cost, Outsourcing Rate, Cost Performance Indicator (CPI), Schedule Performance Indicator (SPI), Uptime, Average Downtime per Month/Year, Machine Downtime Rate, % Planned Maintenance, Number of Releases, Running Cost, Number of Bugs, Number of Pull Requests, Capacity Utilization, Memory Usage, Requests Per Minute (RPM), Errors Per Minute, and others.
All of these metrics don’t make much sense without context. Adopting them from one company to another can be tricky and disruptive, and not always the right way to generate growth. Read Brian Balfour takes on the danger of benchmarks traps.
Related publications:
Thanks for reading, everyone. See you in October!