Retention is one of the hardest problems in analytics. I’ve already spent months breaking down the different types of retention, how to interpret retention reports, and how to choose the right retention logic for your product.

Today, I want to talk about how we actually measure retention - the tools, methods, and data foundations that make retention analytics possible. I’ll walk through my approach to setting up the data layer, SQL, and building consistent retention reports.

If you are new to retention, start here - Refresher on Retention and How to measure cohort retention.

How to report retention

I usually build 3 types of retention reports, each with its own purpose:

1. A consolidated retention KPI: A single number that reflects overall business health.

2. Period-based retention: Segmented by key milestones (W1, M1, M3, M6…) to track trends over time.

3. Cohorted retention: Measuring how each signup or subscription cohort performs.

Each report relies on a different data structure and retention logic. They won’t (and shouldn’t) match, because each is designed to answer a different question.

Let’s break them down.

Read more

This month, paid subscribers learned about:

• Upgrades and Downgrades: Where Most Reporting Goes Wrong - How to measure and report customer upgrades and downgrades, how to make winback reporting accurate, and how to segment all types of returners.

• Rethinking A/B Testing for B2B and SaaS - Why A/B testing for SaaS and B2B is different from experimentation in B2C. Lessons from StatSig on best practices for designing and running experiments in B2B.

• Analysis for Optimal Cadence and Frequency - Applying data science to marketing analytics: how to identify the right frequency for upsells, ad impressions, or notifications.

  Subscribe now

Before we dive in, a quick announcement:

On September 25, StatSig is hosting the StatSig Significance Summit - a conference for product builders in San Francisco, where product and data scientists come together to share their learnings. Speakers include experts from Figma, Grammarly, Atlassian, Anthropic, Lift, Linear, Notion, and more.

I’m partnering with StatSig this year and excited to give away a few free tickets to my readers for the in-person event ($250 value). Email me if you would like to attend!

Back to product analytics.

My first introduction to product analytics was published almost 2 years ago, when I covered why the field emerged, why demand for it remains high, and what skills and qualifications are needed to enter product analytics.

Today, I’m taking it a step further and diving into the projects, tools, and frameworks we use in product analytics - how to break down the user lifecycle, how to approach analytics for different products, and what the market landscape of product analytics tools looks like.

Intro to product analytics - a quick recap

Start here - Inside Product Analytics: Decoding User Behavior

User behavior tracking tools have been around for years, but modern event-based approaches have shaped a distinct domain - and with it, a new type of analyst. An analyst who understands how users interact with the product, how they move across screens, pages, funnels, and features, and what drives them to return or upgrade.

Today’s product analytics tools are event-driven, but events only matter with context. Your success as an analyst depends on creating and maintaining that context - knowing how the product generates data, what’s tracked on each screen or feature, and how to interpret flows, funnels, loops, and trees.

Being a product analyst is not just about managing event data. Product analytics is still analytics at its core. You need the same fundamentals: understanding business strategy and metrics, statistics, critical thinking, data collection, reporting, and dashboards.

You need to know which metrics best measure a product initiative, and how to translate shifts in sensitive metrics into real changes in business KPIs. A product is a subset of the business. So our day-to-day work is about establishing checks, validations, and baselines to ensure numbers are directionally accurate and tell a story.

Product analytics is centered around the product.

Unlike marketing, business, or finance analytics, product analytics is grounded directly in the product itself. Product analysts are fully immersed in understanding how the product works.

A product can be an app, a website, a browser extension, a hardware device, a feature, a piece of software, or a model - essentially any type of application. The better you understand how the product is built, the more effective you’ll be as an analyst.

Easy to say, but in practice, this means there’s a steep learning curve. Transitioning from supporting a SaaS feature to owning onboarding for an Android app is not something you can get up to speed on overnight.

For example, if you support a mobile app, you need to know whether it’s native or cross-platform, which SDKs it uses, how data is collected, how StoreKit works, and how tracking is implemented. And it doesn’t stop there. You must also understand how the app stores work, the technical differences between Apple’s App Store and Google Play, and the full lifecycle of an app version. Mobile stores come with their own tools and metrics. You need to know the types of monetization products offered:

There’s a big learning curve with mobile app analytics.

But one of the most difficult products I’ve ever supported was VidIQ, a browser extension and platform for YouTube growth. Extensions are very different. If you have ever worked with extensions, you must know - users resist adopting them at first, but once they do, they often stick with them for a long time. The user dynamics in an extension are very different from web or mobile apps. There are no standard user flows or navigation patterns to borrow. Retention, onboarding, and usage don’t fit into any benchmarks or proven frameworks. Every A/B test or feature adoption behaved differently on the extension compared to the app. That was frustrating. We couldn’t apply the same learnings or assumptions.

Anyway, every product type, whether a mobile app, web app, or extension, brings its own unique challenges. Since analytics success depends directly on how the infrastructure is set up, as a product analyst, you need to understand how the product is built, how tracking is implemented, and how analytics is generated.

How to get there:

1. Learn to read documentation. Most analysts chase speed and skip onboarding walkthroughs. We don’t take the time to go through “getting started” guides or README instructions. But we should.

2. Be curious and active in the community. Every open-source tool comes with a Slack group or forum. Most tools host office hours, webinars, and publish newsletters. Use the help desk, talk to support teams, ask questions, and seek help.

3. Attend events or watch recordings. There’s a reason I cover events, share free tickets, and write recaps in my newsletter. For example, I wouldn’t know that Apple now offers winback offers (driving up to 20% more paid subscriptions from resurrected users) if I hadn’t watched WWDC. Or that for SaaS A/B tests, you must randomize at the organization level, not the user level, if I hadn’t attended Data Council.

These are random examples, but the point is: yes, it takes effort to read, watch, and filter through the noise. You’ll only use a fraction of what you learn. Still, being a few steps ahead of your peers does make a big difference.

The product is centered around the user.

Let’s talk about the kinds of questions and projects that shape product analytics.

Any type of product (an app, an extension, a feature) is designed for a user. That user might be a human, an agent, a model, or even another piece of software. A user can also be a customer, a consumer, or a seller.

That’s why I approach product analytics from two angles: user lifecycle and personas. Most projects I work on start with the user lifecycle, which would be broken down for every product persona.

User lifecycle

Most of our projects in product analytics are centered around a specific stage of the user journey. For example:

Signup → Onboarding → Activation → Monetization → Retention → Winback

Of course, this lifecycle looks different depending on the product:

• Mobile apps don’t really have “signups.” The journey starts with a download. A signup might represent the completion of onboarding steps, or it could happen before or during onboarding.

• Browser extensions or products with long or complex integration flows often define activation around a successful installation or completed integration.

• Subscription products and SaaS often simplify retention to the inverse of churn: if customers didn’t churn, they’re considered retained.

And so on.

A product analyst’s responsibility is to set up measurements for each stage of the user lifecycle, translate business KPI into suitable product metrics, and then use those metrics to evaluate product initiatives. For example:

• Monthly retention → into MAU.

• MRR/ARR → into successful transactions.

• Churn → into net new cancellations.

• Subscription renewals → into successful payments.

Examples of user lifecycle analyses

Over the last 2 years, I’ve shared many examples and my frameworks for each stage of the lifecycle:

Signup and Onboarding:

• How to Analyze Onboarding Flows,

• A Deep Dive into Onboarding Flow Redesign Analysis, ​​

• How we optimized the onboarding funnel by 220%

Activation and early engagement:

• Activation Done with Regression and Correlation,

• Why Your Activation Analysis Is Wrong - And How to Fix It

• The Ultimate Guide To Product Features Analysis

• Ranking The Top Used Product Features

• How to Report on DAU/MAU Ratio

• How To Get Average Logins Per User Per Day

• The Frequency Of User Activity: SQL and Analysis

Monetization:

• Freemium vs Free Trial Analytics

• Subscriptions reporting: Trials and TTP

• What Is the Optimal Free Trial Length?

• Why Trial Success Rate Is the Hardest KPI To Accurately Report

• Predicting LTV with ML

• An introduction to ARR | CJ Gustafson

Retention:

• How to measure cohort retention,

• Measuring Non-Cohorted Retention or Blended Churn,

• The Ultimate Guide On Churn Reporting (And Its Technicalities)

• A Deep Dive Into User Engagement Through Tricky Averages

• How To Measure Your Adjacent Users

Winback

• Subscription Upgrades and Downgrades

• Upgrades and Downgrades: Where Most Reporting Goes Wrong

• Re-Subscribers: Are Your Customers Coming Back?

Understanding user actions remains a challenge

To be clear, the challenge is not capturing user actions, but reading how these actions come together to form a behavior. And then, understanding what that behavior actually means.

There is no shortage of product analytics tools

But there is a shortage of analysts experienced with these tools

Some tools are better for smaller companies, others for enterprises. Some offer freemium plans or are open source. The most successful product analysts are familiar with many of these tools and can advise product teams on which tool is best suited for a given user volume, revenue milestone, and team plan. Product analysts should know which of these tools are event-based or session-based, whether they sit on top of client-side or server-side data, and which approach is more suited for a team and their reporting.

I plan to do a deep dive into each of these tools over time, though it may take a while to cover them all. I’ve also noticed a few are missing from the map, like LiveSession and Plausible.

What it takes to be a successful product analyst.

The job market is full of analysts proficient in SQL and Python. There are fewer who are grounded in statistics, and when they are, they often move toward ML engineering or NLP rather than analytics itself (very unfortunate). Even fewer have hands-on experience with modern product analytics tools, know the best practices inside and out, and can deliver quickly.

To deliver and bring value, you have to learn:

1. How the product is built, how tracking is implemented, and how analytics is generated.

2. Who the user is, what their lifecycle looks like, how teams expect them to behave, and how they actually do.

3. What the product analytics tool landscape looks like: who the leaders are, who challenge them with bold new features and vision, and which tools are more niche but better suited for specific domains.

I keep saying it: product analytics is the most exciting and fascinating discipline. It’s impactful, and demand for it is accelerating. We need stronger product analysts, we need them now - and I hope my newsletter inspires you (or will inspire you) to become one.

Thanks for reading, everyone!

Related publications:

• Inside Product Analytics: Decoding User Behavior Part 1

• Applying Statistics In Product Analytics

• Applying ML in Product Analytics

• How To Set Up Analytics for Web and Mobile Products

• Anticipating 2025: Top Trends in Analytics

• KPIs Done Wrong: Fixing Common Reporting Mistakes

• How To Measure New Feature Adoption

• How to use product data to drive user engagement and retention

In my experience, no matter how much effort you put into improving attribution measurement, you still end up with Meta over-reporting trials by at least 3x, and somehow marketing channels reporting you more signups in November than you actually received in the entire Q4.

It’s ironic - product analysts work with low-trust data, while marketing analysts have to work with blatantly over-reported data from every source. Literally, you have to divide every metric by 7 just to get it somewhere close to reality. No, thank you.

Anyway. Despite me not enjoying marketing analytics and advocating for growth and pricing to be owned by product teams, we should still be skilled to run analyses and support poor marketing folks as best we can. That means knowing the frameworks and types of analyses around ad ROI, email cadence, push notifications, payment upsells, and more.

So today, I want to resurface my long-overlooked framework for using ML to determine the right frequency of upsells, ad impressions, and different types of app notifications - how to find the frequency that drives the highest DAU, conversion, or sales without harming (or at least acceptably impacting) user engagement, unsubscribers, or churn.

Regression vs. Classification: choosing the right ML

Some time ago, I shared a guide to help you decide which ML model to pick to solve a problem. I pointed out that regression is the most common ML we use to answer many product and BI questions, including LTV predictions, revenue forecasts, estimating the number of new page views needed to improve signups, how many notifications increase DAU, and so on.

However, linear regression isn’t the right method for determining the threshold between multiple outputs.

While regression is the right ML to apply to understand the relationship between a metric and user actions (and how much change in one variable affects another), this is not the right approach for finding the right threshold or cadence. For example, measuring a variable (e.g. emails, notifications, ads) against 2 or more output metrics (like DAU, activations, CVRs, unsubscribes, churn, etc) is not a regression task, and the best method here would be to leverage (a) classification models and (b) testing.

Before we dive deep into that, a quick reminder on regression:

• Use a simple regression model with a single independent variable:

  • screen view → activation

  • campaign impressions → trials

• Use a multiple regression model with 2+ independent variables:

  • screen view + age → activation

  • upsell click + blog opt-in + referred → churn

Remember, multiple regression is designed for multiple input values, not output metrics. So, you can’t apply it to estimate clicks and unsubscribes at the same time. And that’s the reason why classification models would fit better for this task than regression.

Methods and steps to estimate the right threshold

Below, I share 2 approaches to locating the right frequency or cadence:

1. A quick “back of the napkin” analysis when you have a few hours to offer an estimate.

2. A multi-label classification model, when you hopefully have at least a week to classify users and develop a model to predict the likelihood of users clicking on the upsell or ad and not opting out or churning.

Read more

Today, I am back to measuring user engagement, as promised.

This publication is long overdue for a deep dive into DAU/MAU. I’ll re-introduce the DAU/MAU metric, clarify common misconceptions about its purpose and usage, and walk you through examples of how to report it right, along with mistakes to avoid.

What is DAU/MAU, and why do we need it?

The DAU/MAU Ratio is the most common measurement of user engagement. It shows what % of monthly users interact with your product every single day. In other words, how many users engage with the product each day in a month.

Read more

This is a follow-up to my (now very old) Introduction to Event-Based Analytics guide, where I did my best to squeeze in 11 weeks of digital analytics course into a single short newsletter.

Today, I want to take it a step further and share my framework and recommendations on how to set up analytics that will scale up, allowing better event tracking for faster product testing and iterations.

Here are a few “groundbreaking” truths about setting up and maintaining event-based analytics:

You need a dedicated owner for event setup and maintenance

While maintaining events requires cross-functional effort (product, analytics, development, and QA), the ownership of the tracking setup and the responsibility for updating the taxonomy, data catalog, and documentation should be assigned to a single role. This role can sit within either the product or data team.

Distributing this ownership across PMs, analysts, or multiple teams slows everyone down, leads to overlapping responsibilities, redundant workflows, and ultimately introduces more gaps in your analytics.

The more experienced this person is, the higher the ROI of your analytics setup. Especially in the early stages, you want someone who understands how to report on the Signup-to-Paid rate, replicate it in a funnel, and knows how to link the conversion to one user, not simply divide the 2 events in the formula.

📌 Related: I’ve been training 3 standout analysts over the past year in event data setup, kicking off taxonomies, and event data maintenance - happy to share introductions if you’re looking for someone!

Event-based analytics ownership is still new and underrated

If your event tracking isn’t trusted, most of your analytics efforts will fail.

Teams are often quick to point out discrepancies for the same metrics between tools or data sources, but reluctant to invest in product analytics.

We refer to this role as a product analyst or digital analyst. This unicorn role requires product intuition, an understanding of the product development lifecycle, knowledge of data and data architecture, and most importantly, strong analytical thinking. This person needs to:

Read more

Subscribe now

A year ago, I shared my analysis on Activation I did at MyFitnessPal. Many apps reached out with questions about modeling, performing regression in product analytics tools, and handling tricky Activation cases.

So today, I want to follow up on Activation and show you how to identify a-ha moment, different ways to calculate it, and what to do when you’re not getting a clear signal or your data is inconclusive. I’ll also walk you through how to get Activation in product analytics tools and validate it through analysis.

If you’ve been reading any PLG-related content, you’re likely familiar with Activation or a-ha moment. I believe it initially gained popularity with Facebook’s famous “7 friends in 10 days” and was later revived with the PLG movement.

Today, I won’t be discussing the concept of Activation but rather how to calculate it using product analytics tools or modeling. If you are new to Activation, please start here:

1. Activation: The Product Metric Everyone Thinks They Need But Can’t Seem To Define

2. Rows: How to pick your activation metric

3. Crafting The First Mile Of Product

4. What is a good activation rate

Locating Activation milestone in product analytics tools:

Here’s what 90% of growth marketers do to locate Activation:

1. Create an onboarding funnel that tracks every onboarding step.

2. After the last step in the funnel, add one more step for the first successful key action users complete (e.g., create a report, listen to a song, log an exercise, etc.) and call it the Activation rate.

3. Set the funnel to 1, 4, or 7 days.

4. Done.

This is an incomplete and inaccurate activation funnel. However, it’s the best you can do using the Funnels feature in product analytics tools.

Then, analysts step in and create a cohort of returning users who completed the key action after returning within X days. We do this using Retention or Heatmap features - not the Funnels.

But this still isn’t Activation. It’s simply the % users performing the core action on a given day. It might indicate Activation - or it might not. And there is no way to definitively know unless you leverage Compass in Amplitude or Signal in Mixpanel. However, both come with data context and caveats. Let’s break it down.

Calculating Activation - Expectation vs. Reality

Expectation:

You open Amplitude and instantly see your Activation milestone clearly within seconds:

Reality:

Such analysis worked for MyFitnessPal because (a) I spent months setting up and fine-tuning analytics there - we had to reset events, remove noise, configure retention definition, and more, and (b) we had 10 years of data.

If your app was launched recently and your analytics weren’t set right, your Activation report is likely to be inconclusive.

It doesn’t mean your product doesn’t have an Activation moment. It just means you’re not capturing it correctly. The missing signal could be due to noise in event tracking, outliers skewing data, insufficient data, skewed distribution, or other data inconsistencies.

If this happens, follow the steps below to manually find your Activation milestone using product analytics tools. Whenever I feel lost in data or struggle to capture the signal, I use this framework - and it works every time.

How to do Activation analysis

Step 1: Start with the most trusted event for a product feature

Begin with a clear, well-defined “action” event that signals when the activity is completed, such as transaction_completed, food_logged, report_submitted, etc. Ignore less meaningful events like views, clicks, toggles, banner views, etc. These often introduce noise rather than actual user engagement.

Step 2: Create cohorts based on feature usage.

Define cohorts based on specific feature interactions, such as Users who logged foods at least once, Users who logged exercise at least once, Users who integrated a new device at least once, etc.

Each persona should have its own cohort.

For a typical app, you’ll likely create 10-12 cohorts. Each cohort must be modeled separately. This is a lot of work. It won’t be done in a day.

Step 3: Create 2 reports for every cohort:

For each cohort, generate 2 key reports:

1. How many times users completed this particular action (once, twice, 5 times, 10 times, etc.)

2. When they completed this particular action (Day 0, Day 1, Day 2, etc.)

Step 4: Define the retention metric for Activation.

I don’t use Retention based on built-in Amplitude or Mixpanel values like New User, Any Event, or Any Active Event. If possible, my retention is narrowed down to a very particular returning action, like reading a book, logging food, submitting a request, or completing a transaction, etc.

However, for some apps, a simple app open or screen view event may work just fine. The key is to select a high-level event that users can do on every return but is still well-defined and meaningful.

Step 5: Run correlation analysis

For each of the reports for every cohort, run a correlation analysis comparing feature usage with multiple retention timeframes: Day 1, Day 7, Day 30, Month 3, Month 6, and Month 12. That’s a lot of plots like these:

This is where most people stop.

This is NOT Activation yet :). But we are getting closer.

Step 6: Run regression modeling for the key 2-3 patterns

I described the steps and process here - How to do linear regression and correlation analysis.

What’s important to understand is that Activation is predictive of user retention. Whenever you see “predictive,” it means you need to either model it or run an A/B test.

For modeling, it doesn’t need to be complex ML. In the example above, I used linear regression, and it worked well for me. Linear regression takes correlation analysis further and shows how much one variable affects another and whether you can use the pattern of one variable to predict and estimate the behavior of another.

Linear regression doesn’t prove causation, but it does give you more confidence that there’s a strong connection between variables and how this connection is expected to change if you increase or decrease variables.

For this step, you can use this quick template - Correlation and Linear regression template. Run regression analysis on the 2-3 strongest patterns from your correlation step. Read Decoding Regression Scores to understand how to interpret regression output. The regression model with the highest R-square value is your Activation milestone. Congratulations - you did it!

What if it still doesn’t look right?

Sometimes, even when you follow best practices, the data doesn’t return the expected output. Or it doesn’t make sense.

• Check if your data is significant. This is the most common cause of flawed analysis. You may not have enough data for certain actions, even if they are core features. If this is the case, you have to work with qualitative data - use user surveys and research. You can’t model Activation just yet if the data isn’t there.

• For subscription apps, measure Activation against paid renewals. Do NOT measure Activation against just any retention metric in Mixpanel. Activation should be tied to paid renewals for each plan you offer. If your app is new, you may not have enough renewal data, especially for annual subscriptions. In this case, you may try your luck with a proxy metric - use activity data for paid users.

Thanks for reading, everyone. Until Wednesday!

Subscribe now

In a very short period of time, the number of experts discussing product onboarding and activation has surged. Some argue that onboarding flows should be long enough to give users a chance to fall in love with your product, as longer flows attract high-quality users. Others say that onboarding should be quick, no more than X steps. Or absent at all.

Everyone seems to have an opinion on onboarding flows. However, no one explains how onboarding flows should be analyzed and reported.

Here is the thing: product analytics tools don’t support complex onboarding flows. They simply don’t. Yet, most onboarding journeys today are complex. Many products are cross-platform, with web-to-app transitions, dynamic “flying” paywalls appearing randomly between onboarding steps, GDPR screens for certain users, multiple sign-up methods, personalization, and more.

To my surprise, most onboarding flow metrics are proxies or represent only a small subset of the flow that teams can track. Yet, this doesn't stop today’s growth experts from speculating or making recommendations about the “right” or “wrong” onboarding flows. Or worse - testing them.

Today, I’ll walk you through examples, steps, and methods for analyzing complex multi-step user onboarding flows, with or without product analytics tools. I’ll also share case studies on how to report and interpret onboarding data.

Read more

Subscribe now

Over the years, I've shared a lot about retention, but it's scattered across different sections or featured in other newsletters where I was a guest writer. As I am now onboarding 2 new analysts on my team, I was looking for a consolidated guide on retention, rather than sharing a list of 40 articles from my newsletter.

So today, I want to consolidate the must-know things about retention - its definition, best practices in reporting, examples of reports, and key points to remember - into this single publication.

10 reasons why working with retention is difficult

1. Retention KPIs include multiple variables such as - signups, user activity, often with filters, and varying date ranges. You often can't finalize reporting until all other metrics are in place - Top of funnel, transactions, DAU, etc.

2. That’s why, creating and automating retention reporting takes time, and maintaining it requires effort.

3. It is an eco-system metric. Retention isn't directly actionable or sensitive. You need to break it down to make sense of it or take action.

4. There are many types of retention, and no single “right” method exists. Each company defines and calculates retention in its own way.

5. It’s actually wild out there: some do unbounded, others N-day, or cohorted, non-cohorted, blended, adjusted, weighted, etc.

6. This makes using retention benchmarks absolutely pointless!

7. Every new executive brings their own definition of retention, often changing it to adjust to a specific cohort or activity every few months. And you were wondering why I don’t like semantic layers.

8. To make retention actionable, you have to link data to specific marketing campaigns, product launches, or A/B tests.

9. Certain types of retention (unbounded) can’t be used for MoM or YoY growth analysis. You will have to maintain multiple versions of the same metric, introducing more confusion for the teams.

10. I've never had luck with retention reporting from analytic tools such as Heap, Amplitude, Mixpanel, RevenueCat, and Chargebee, and I often have to replicate reports myself to ensure they are accurate and trustworthy.

Retention reporting is different for B2C, B2B, and SaaS

Retention for B2C has a different meaning and objective than it does for enterprise, B2B, or SaaS companies.

Retention for B2C (Calm, MyfitnessPal, YouTube, eBay)

Read more

Subscribe now

I came across an article the other day on the Mixpanel blog titled Why PMs should be responsible for revenue metrics. Umm… what else should they be responsible for? Sorting color stickers in Figma?

At first, I laughed. But then I remembered how the product manager on the team I was supporting (an amazing PM, btw) showed me his MRR tracker in Amplitude, which was based on the price passed for the payment_success event. The problem? It included price values for free trials - and, even worse, it counted both trials and trial-to-paid conversions for the same user as 2 different events in a monthly report. So, he was counting the subscription price values for free trials and then doubling them when those trials converted within the same month. He was over-reporting MRR by at least 70%!

What’s tricky about that case is:

• It’s so common! Using payment_success is a popular workaround today, especially when companies try to bring Stripe data into events— and it’s riddled with caveats.

• If they passed to Amplitude a $0 for free trials and the actual price for paid subscriptions, his report would have been accurate.

• If they had passed a separate event for trial_started and then another for paid_subscription_started, his report would have been accurate.

There are a lot of ways this report could have made more sense— and even more ways where it doesn’t. This often falls outside of a PM’s scope or visibility. So, should they really be responsible for creating revenue metrics, after all? Even if the tool makes it super easy?

So today, let’s talk about all the things we get wrong with KPIs - common mistakes with  KPI definitions, formulas, logic, and the pitfalls of reporting. 

We talk about metrics and KPIs way too much.

It’s true. Every article or guide out there talks about the importance of choosing the right metric to measure business health, warning that if you pick the wrong KPI or formula, everything will go downhill.

When things go bad, it’s visible everywhere. How bad? That’s for analysts to figure out. But you can’t hide the elephant in the room. If your transactions or subscriptions don’t grow alongside a 3x increase in traffic or leads, no KPI can mask it, no matter how it’s calculated or defined.

Plus, no one hands you the luxury of picking the KPI or North Star. Those decisions are made, over-debated, and agreed upon long before you’re involved - unless, of course, you’re part of the founding team.

Following benchmarks too closely

First, we live in a world obsessed with benchmarks, and I’m convinced they do more harm than good.

Take retention, for example - don’t compare unbounded retention to a bounded benchmark. If the Day 30 retention benchmark is 9%, that’s very likely bounded (or X-Day). But if your retention rate is 14%, it’s likely unbounded, so it’s not an apples-to-apples comparison.

I love quoting Brian Balfour here from his take on Growth Benchmarks Are (Mostly) Useless:

“Different businesses measure the same metric completely differently even if they are in the same industry category. I’ve never seen a benchmark report that takes this into account. They usually just ask, “What is your CAC?”  

Different products and business models require different ways to measure customer acquisition cost, and other key metrics that often show up on benchmark reports as uniform. 

Averaging or lumping together CAC can be extremely misleading because it doesn’t take into account your company or product’s specific business model. For example, if you have multiple tiers in your SaaS product, average CAC is a lot less actionable than CAC sliced by your different customer segments (with each segment paying different subscription fees).”

Borrowing KPIs from other products

I love this example from : here are 20 ways to calculate ARR: 

I’ve gone through 10 different definitions of DAU over the last 4 years— and most of them weren’t perfect: 

People switch companies and bring their frameworks (based on specific metrics) with them without considering whether a weather channel success stories fit a nutrition tracking app. It doesn’t.

Some common mistakes senior analysts often make:

• Bringing SaaS reporting into B2C subscription analytics. For example, ignoring trials or Trial-to-Paid CVR in subscription reporting. More on it here.

• Using B2C transactional metrics in B2C subscription products, such as including not paying users in LTV or ARPU calculations or adding new users into churn metrics.

• Bringing SaaS or B2B metrics into B2C, such as using churn to measure product abandonment or trying to measure inverted retention.

Each product type requires tailored metrics and frameworks that reflect its unique business model and user behavior.

Using the same KPI for different use cases

Every metric has its own purpose - whether it’s ecosystem, secondary, tradeoff, proxy, success, North Star, OMTM, etc. Developing (and maintaining) multiple variations of the same KPI and ensuring teams use it appropriately across different contexts is a real challenge.

For example, here are a few contexts on how to report churn:

So, theoretically, I’d need to create at least 7 different calculations for one KPI, depending on who’s using it, when, and for what purpose. Good luck with semantic layers! And metrics trees! Let me know how that goes 😈

📝 I often see people debating the “right” way to calculate a KPI, arguing over which method is more accurate or appropriate - like whether to use bounded or unbounded retention, whether to exclude new users from DAU, or whether to exclude canceled trials from churn rate. The reality is, we actually need all of these KPI variations to fit different use cases.

Over-complicating metrics

I’m definitely guilty of this. As I mentioned earlier, when sensitivity is at play, you need metrics to be precise. You want to exclude edge cases like double-firing events, problematic timelines where data was affected, test users, etc. But chasing reporting accuracy often comes at the cost of making it more complicated.

For example, here’s the actual DAU definition I had to come up with for a productivity-tracking app:

DAU = session start (with session length capped at 45 min) + login (excluding unauthenticated users) + onboarding page view (excluding unauthenticated users) + CTA (login status = on) 

It was super accurate and trusted. But replicating it across 5 data sources? Yeah, that was fun.

Another educational app I’m working with right now has many different features spread across the app, and their DAU was defined based on 11 unique events - some with additional filters. I can’t believe it’s been 4 months already trying to switch their reporting to a simpler approach while maintaining the same activity milestones.

It’s all about finding the balance between precision and ease of report maintenance. If you can, aim to keep KPI logic simple and intuitive. Limit the number of variables and calculations. This will help you:

• Replicate the KPI across multiple platforms.

• Reduce the probability of errors.

• Facilitate iterations—because you’ll likely need to adjust your KPIs every few months.

So, don’t listen to everyone. Just do: DAU = app open + signup. You can thank me later.

Using the wrong proxies

Activating more streaks or sending friend invites doesn’t necessarily serve as a proxy for LTV or revenue. You need more analysis to prove the relationships between sensitive metrics and ecosystem metrics.

Examples of proxy metrics:

• DAU: Unique screen view, Login, App open, Median time spent per user per day

• Activation: Onboarding CVR, % users doing X activity in their first day.

• Retention: Total screen views, Logins, App opens, Unique days users using the product/feature.

• Install-to-Paid: Signup-to-Trial, Total trials, Number of initial transactions.

• Churn rate: % users canceled, Total number of cancellations, Median time spent per user per day.

• LTV: DAU-to-Paid, AVG transactions/orders per user, % paid customers from DAU.

• Revenue, MRR: Total transactions, Paid customers, Upsell-to-Trial CVR, AVG revenue per user

How to determine if your proxy metric is effective:

1. The proxy metric should be sensitive enough to be influenced in the short term - such as screen views, button clicks, session time, or number of transactions.

2. Good proxy metrics are simple and don’t involve many complex filters or calculations. 

3. The metric should be independent of other product features, marketing initiatives, or similar factors.

4. The change in the proxy metric should indicate the direction of change in the target metric.

Learn more about proxy metrics: Introduction To Proxy Metrics and How to find optimal proxy metrics.

Losing Proportions between KPIs

In last week’s publication, Mastering Critical Thinking: How to Improve Your Analytical Skills, I introduced the concept of measuring compounding: 

• A 25% increase in signups leads to a 1% increase in trials.

• A 1% increase in trials translates to a 0.5% increase in paid subscriptions.

• A 0.5% increase in paid subscriptions results in a 0.02% increase in revenue.

How do we calculate these? 

It starts with understanding the ecosystem of metrics. All KPIs and metrics are interconnected. If one metric increases, you need to know which other metric may decline and by how much. That’s why we rely on waterfalls and funnels to measure compounding factors.

Examples of subscription metrics relationships:  

• High churn = low retention. The higher the churn, the lower the retention, and vice versa.

• Total transactions = New subscriptions + Renewals.

• Net new subscriptions = New subscriptions - Churn.

Examples of engagement metrics connections: 

• If 30% of users engage 2+ per week, the DAU/WAU ratio should be at least 60%.

• If the WAU/MAU ratio is at least 60%, DAU/MAU should be over 20%.

• If 40% of users activate the product during the first 24 hours, then R(Day 1) should be at least 50%.

And so on.

As I mentioned last week, in analytics, it’s rare to see a report that clearly states, “+20% increase in Signup-to-Trial CVR since 9/15/2024 attributed to X campaign.” If that were the case, we’d probably be out of a job.

Instead, what typically happens is this: Next to a +20% increase in Signup-to-Trial CVR, you might see a (-10%) decline in Signup-to-Paid CVR, a (-12%) drop in total trials in Amplitude, with Meta showing trials are up 560%, while GA is reporting flat signups, and Snowflake’s Downloads data is incomplete.

That’s why maintaining metric proportions and having clear baselines is essential. Even if you have low trust in reporting or things don’t seem to add up, understanding the movement of one metric can help you interpret the movement of another - bringing some much-needed confidence to your report.

Takeaways

1. Don’t overthink KPIs. If you're unsure how to measure an initiative, stick to simple metrics like unique views, CTA clicks, and the % of users with a CTA click. Not everything needs to be tied to LTV or MRR.

2. Use an effective proxy metric that is both sensitive and independent. If your proxy requires complex calculations, it’s not a good proxy.

3. Don’t stress about benchmarks - focus on your Signup-to-Paid MoM growth.

4. Avoid bringing metrics definitions from your previous job into your current project. Every product is unique, with different user lifecycles. Some apps have monthly and annual subscriptions, while others have 18 payment plans. Churn calculations will vary. Develop KPIs that fit this specific business and product.

5. Keep metric logic simple by minimizing the number of variables and calculations.

Thanks for reading, everyone!

Related publications:

• Introduction To Event-Based Analytics

• Inside Product Analytics: Decoding User Behavior

• How to measure cohort retention

• How To Do a Root Cause Analysis

• Ranking The Top Used Product Features

• Subscription Upgrades and Downgrades: A Deep Dive into B2C vs. SaaS Reporting

• How To Measure New Feature Adoption

• The Ultimate Guide To Product Features Analysis

• How to Communicate Data Effectively

• How To Find Optimal Proxy Metrics

• Measuring Non-Cohorted Retention or Blended Churn

• How to use product data to drive user engagement and retention

Subscribe now

Analytics is more than SQL queries or knowing how to calculate key metrics. At its core, effective analysis is about framing problems, drawing insights, and making informed decisions based on critical thinking.

This type of thinking is difficult to develop, especially if you are self-taught. It takes time and practice to train your mind. 

This is the main reason why many skilled analysts fail with technical challenges during interviews. Not because they don’t know how to write SQL, calculate LTV, or sort arrays in Python, but because they are distracted with details, pick the wrong algorithm, and (most commonly) miss signals provided in their task or input data.

This publication continues from Introduction To Problem-Solving And Critical Thinking, published last year. If you're new here, I recommend starting there to learn: 

1. The basics of problem-solving and informed decision-making.

2. How to develop critical thinking and strengthen your analytical intuition. 

3. The lifecycle of analysis - its steps and process. 

4. Common methods and types of analysis. 

Today, I want to take you behind the scenes of analytics, walking you through methods of critical thinking and showing how to integrate them more into day-to-day projects. 

Since my previous piece on problem-solving was more “theoretical,” today I’ve grouped concepts into the top 10 easy-to-understand techniques to improve your analytical thinking. I’ll keep it short and light.

Most importantly, I want to reiterate 2 key messages:

1. Your success as an analyst depends on your analytical intuition — nothing else matters. PMs, data engineers, and other roles transitioning into analytics often struggle, regardless of their SQL skills or understanding of event relationships, because they lack analytical thinking.

2. Contrary to popular belief, analytical intuition can be developed and improved. Below, I will show you how.

Read more

Subscribe now

Understanding the frequency and volume of product usage is the most important step in unlocking insights into user engagement and behavioral analytics. We talk about DAU and retention all the time (I’ve probably written over 100 articles on engagement analytics alone), yet working with event data still requires time and effort to make sense of events and find the right way to structure them.

Last week, I introduced a new template for a Mobile Subscription App Engagement and Retention Dashboard for Amplitude users. The dashboard tracks metrics such as DAU, the DAU/MAU ratio, average usage per user, app usage frequency, month-over-month views, and more, offering a comprehensive view of user engagement and lifecycle. 

In this publication, I’ll walk you through my template and top engagement metrics, showing how to identify daily or weekly app usage cycles and how to report top actions and retention.

If you use Amplitude, you probably saw last week’s announcement - Amplitude Made Easy - where they introduced “effortless” (sure, we’ll see) analytics via expert-created templates. I am honored to join renowned PLG experts like Kyle Poyar, Elena Verna, Dan Schmidt, Drew Teller, and, of course, :

You can browse through their Template Gallery, pick a template, customize it, and make it your own. I’ve reviewed most of them, and each one offers something unique. They’re all helpful, but obviously, mine is the best 😈! After all, nothing is more exciting than uncovering product usage patterns. When you understand which features and user milestones drive engagement and retention in your app, everything else falls into place like a puzzle.

Top 5 Engagement KPIs

I like to open each of my dashboards with a bold display of the top 3-5 KPIs that clearly represent the engagement ecosystem:

Right there, we see the most important engagement milestones (DAU, WAU, and MAU) and their distribution: 

• 85% of users engage with the app every day of the week.

• 60% of users engage with the app every day of the month.

This view of the top 5 metrics is designed for executive teams, who (hopefully) have the dashboard bookmarked. They can quickly open it during calls to check a baseline or a benchmark.

For my team and myself, I prefer a timeline view of each metric segmented by platform or top regions, like this:

📌 Depending on how you define an "Active User" (whether you include new users or not), you should set user filters as New or Active.

📌 For mobile apps, no matter how precise you are in defining Active Users and filtering out new users from DAU, some % of DAU will still be categorized as New Users (due to re-installations, different sign-up emails, or use of different devices, etc.).

Reading charts above:

• The first DAU chart highlights natural user cyclicality, with activity dipping on weekends. This can help guide the timing of user outreach, such as notifications, emails, or feature announcements.

• The second WAU chart is useful for (indirectly but accurately) monitoring the success of ads and marketing campaigns.

• The third MAU chart illustrates seasonality and external factors influencing user behavior. You can take it a step further by segmenting MAU into cohorts such as new, core, or resurrected MAU to gain deeper insights.

More usage doesn’t mean better usage.

When analyzing engagement data, it's crucial to monitor activity spikes. An increase in DAU doesn't always translate to better feature usage or improved retention. Here are three common types of spikes: 

• Top-of-Funnel Spikes: These occur when a campaign goes viral or ads are boosted, leading to a surge in new installs. This causes sharp increases in DAU, WAU, and engagement metrics across the board (driven by new users). However, this spike often declines over time, bringing your net new DAUs back to close to zero.

• Notification Spikes: Teams often experiment with notification frequency to re-engage inactive users. After sending notifications, you might notice a temporary DAU increase. However, this boost, unfortunately, often doesn’t translate to long-term MAU growth. If this is the case, follow the user journey to identify when and why users lose interest.

• Start Session spikes: When a new feature, screen, or menu is added, Start Session events may rise, resulting in an overall DAU increase. However, this spike can sometimes be due to improved session tracking (or, worse, background activity being captured), creating a misleading view of increased engagement. (I covered it more here - Do You Over-Report DAU?)

This is why, alongside top engagement metrics, it’s also important to track the frequency of usage and averages:

Frequency of usage: The chart on the left is one of my favorite views. It shows how frequently users open your app on a given day, week, or month (this example uses a monthly view).

1. This is not real data. It shows an unusual usage pattern: many users engage with the app over 100 times per month - something rare. Typically, most users fall into the 1-3 usage bucket.

2. As your app matures, it’s exciting to see how these user groups shift to higher usage buckets. That’s how you know users love your product - they use it more than 5 times per week or more than 20 times per month.

Average activity per user: The chart on the right displays the average activity per user, segmented by platform. Usage patterns tend to vary significantly between web and mobile. I’ve set it to an overtime view to see a histogram. However, this metric rarely fluctuates unless a new big feature is introduced.

❗Remember: Whenever you see an "average" metric, watch out for outliers.

❗Also, if you have a large number of test users, bots, or admins impersonating a group of users, this will be reflected in averages, and hopefully, you have a way to filter these out.

Average activity per user is not sensitive, so if it starts to decline, it’s a major red flag. Such a drop will likely signal an upcoming retention decline. You probably missed early warning signs such as decline in onboarding flow, activation, paywall views, or screen views. If you are dealing with average activity per user decline, you should segment DAU into cohorts (New DAU, Core, Power, Resurrected) to pinpoint which user persona is underperforming and at which product milestone.

Monitoring product feature usage 

I often refer to product milestones, which include the onboarding flow, activation, and repeated core app usage. For tracking core feature activity, I use these charts:

In my template, I don’t explicitly label anything as “activation” and instead refer to it as “main app activity.” This is because pinpointing activation requires a certain level of analytics maturity and a significant amount of data, which newer apps might not yet have. Instead, I recommend using 2 key metrics:

1. The top actions you expect users to perform in the app.

2. The volume of users who complete these top actions.

For a deeper dive into product feature analytics, read The Ultimate Guide to Product Features Analysis.

I designed my dashboard template to be adaptable for most mobile apps and, hopefully, easy for everyone to read and understand - whether technical or not. I don’t believe there’s a template out there that you can plug in and be done with. You’ll still need to add filters, adjust segmentation, set date ranges, and apply other customizations for any template to fit your needs. So, I wouldn’t call it “effortless.”

That said, having access to a gallery of pre-built visualizations and formats, all vetted by domain experts, is a game changer. You can leverage their metrics and definitions to understand which questions to ask and how to find the answers. It’s incredible to see how far product analytics have come! 🚀📊

Thank you for reading, until next Wednesday!

Related publications:

• Introduction To Event-Based Analytics

• Inside Product Analytics: Decoding User Behavior

• Ranking The Top Used Product Features

• How To Measure New Feature Adoption

• The Ultimate Guide To Product Features Analysis

• How to Communicate Data Effectively

• How To Find Optimal Proxy Metrics

• Handling Missing Data: Should You Drop or Impute?

• The Database of Winning A/B Tests

• Measuring Non-Cohorted Retention or Blended Churn

Every year, the first week of June directs the world's attention to WWDC, an important event for everyone involved in development, data, or analytics. If you touch data in your work, I hope you didn't miss the updates this year that continue to shape and define analytics.

The app world has transformed analytics, leading to the creation of a new domain and what I call a new type of analyst who works with app behavioral data. This role combines product analytics, growth analytics, and BI, requiring a deep understanding of the mobile development lifecycle, how payment stores operate, SDKs, and how users interact with apps - how they navigate between the screens, paywalls, and funnels, and what makes them come back, upgrade, share, or abandon.

WWDC 2024 emphasized Apple's commitment to user-level personalization. In a nutshell, this year’s updates focused on reducing tracking while increasing tools for app control, promotion, and management. Before declaring the end of traditional analytics due to increased privacy, it's worth noting that Apple also introduced several workarounds and shortcuts to gain attribution and insights.

So, let’s dive in - my top updates from Apple that impact analytics.

Read more

Subscribe now

Today, let’s talk about proxy metrics.

A month ago, I published How To Find Optimal Proxy Metrics, in which I referenced a recent research study from Google and Stanford on the importance of using sensitive metrics instead of KPIs for A/B testing. Researchers introduced the concept of Pareto Optimal Proxy Metrics, which significantly optimized the accuracy and sensitivity of lift predictions.

That publication provoked a lot of discussion, and I received many questions about proxy metrics. So today, I want to take a step back and offer a “proper” introduction to the concept of proxies.

What are proxies? Why do we need them, or when do we need them? What are the methods for finding proxy metrics? How do you know when your proxy metric is not an appropriate representation of the KPIs? Examples of proxies and their use cases.

Read more

Subscribe now

One of the most common challenges in product analytics is measuring the success of a new feature release or analyzing product adoption - understanding how users discover it, how they use it, and how often. Simple questions can become quite complex when you start breaking them down by timeline, user segments, or engagement layers.

Measuring brand-new feature adoption is my least favorite topic because:

1. There is nothing to compare new feature performance against - no baseline, no control group, and no easy way to know if the volume of usage you are seeing is good or bad.

2. Often, new features are announced via GTM campaigns that skew the feature's impact and complicate analysis.

In this publication, I want to share what I have learned about new feature rollouts, show how to run analysis to evaluate its usage, and offer metrics for measuring product adoption.

Teams often confuse feature rollout with feature testing. While the steps and procedures for release may be similar, statistics and analysis are different. Unfortunately, not many understand this distinction.

New feature release vs. feature optimization

I prefer to differentiate between experimentation for introducing a new feature vs. optimizing an existing feature and classify all product tests into 3 categories: 

1. Optimizing an existing product or feature:

This is the most common A/B test type. You simply change the color, format, and positioning of a known (existing) feature that doesn’t change the user’s path but is intended to optimize the experience. Usually, these are fast, low-impact tests. It’s rare that you notice a significant conversion change, and most likely, there will be low variance in your results. 

2. Introducing a change to an existing product or feature:

In this case, you introduce a change to a known feature that affects the user journey or path. It may be a test to reduce or add more steps for the signup form, reroute users through different page flows, get to your value pitch by adding/removing CTAs, or anything that introduces a new user path. 

3. Introducing a new feature that didn’t exist before:

This is the hardest type of experimentation and the most commonly conducted and evaluated incorrectly. It should NOT be treated as an A/B test, and here is why:

Read more

Subscribe now

I am excited to share that my new publication was featured on the Amplitude blog yesterday! 

Make sure not to miss it:

The Ultimate Guide to Product Features Analysis

If you are a product owner, product manager, or data scientist working with product analytics, this is a must-read 10-page guide to learn:

• Which product features are the most popular and drive the highest growth and engagement? 

• What are the feature value and ROI of product initiatives? 

• What features are the strongest candidates for optimization, monetization, or paywall?

• And which should be sunsetted?

You need to do a feature analysis to understand your value proposition, appropriately package your pricing, tailor your messaging and GTM strategy, prioritize product initiatives, discover opportunities for new feature development, and simply connect with users more.

In this guide, I offer my framework for doing a feature usage analysis that can be done in popular product analytics tools like Amplitude or Mixpanel. I have used this approach at VidIQ, Change.org, and now at MyfitnessPal, as it’s applicable for most B2C, B2B, and SaaS apps and is easy to borrow and interpret. I use MyfitnessPal and Strava as app examples and demonstrate my analysis in Amplitude. 

I am sharing a few key takeaways below. Please read a full guide here: The Ultimate Guide To Product Feature Analysis.

Why feature analysis isn’t easy

It requires detailed data tracking: what makes feature analysis unique and challenging is that unlike root cause analysis, activation analysis, personas, or activity frequency analysis, for feature usage, you need to have every user action in the app or website tracked, and this tracking should be mapped to the appropriate feature (or a group of features) classification.

It depends on matured analytics: Depending on your product's nature and maturity, this task can be daunting and can require comprehensive documentation, catalogs, or taxonomy. 

It’s not adaptable: There is also no plug-in template or easy solution to fit all features because every product is unique and has its own functionalities.

Getting started with feature analysis: 

Align on definitions of a feature

There is no hard-coded definition of what a feature is or should be, and it’s up to your product team to align on the functionalities you wish to consider as features. It can be a mix of ML-based recommenders, notifications, and in-app screen modes and settings. Some features may not necessarily require proactive user interaction (e.g., dark mode screen view, no-ads experience, reminders). Measuring engagement for such features is very tricky.

Cluster or group your features

Some features can belong to multiple groups or clusters (e.g., in the Strava app, posting an exercise on the newsfeed can be part of the Activation, but it can also belong to the Community features or belong to the Progress tracking). Such features naturally will have more volume of usage, but this does not necessarily mean they are more popular. To address this, you can bucket features into clusters or groups and set a baseline of expected usage for every group to ensure you do not over-count usage. For example, features can be grouped into community, onboarding, core, creator, etc.

Identify transitional features

Users often have to use one feature simply to access another they intend to use (e.g., a dashboard in the Apple Health app can be a feature itself, or it can also be a path to other health reports). You need to have a way to differentiate and, if needed, exclude transitional usage from the actual feature usage. Read a full guide to learn some quick ways how to do this in Amplitude.

Recognize features are dynamic.

Every company iterates and expands its offerings. For example, in the past year at MyFitnessPal, we released many new features, such as Dashboard, Intermittent Fasting Tracker, Glucose insights, Personalized plans, and more, while also continuing testing and iterating. The feature analysis I ran last year is no longer applicable today, unlike my analyses for Personas, Activation, LTV, or Retention. You have to revisit your feature clusters to make sure they are still appropriate, and their tracking hasn’t changed.

Steps for feature analysis: 

1. Create Feature Matrix - a consolidated view of your product features across your app or business. Its main objective is to document and track your value offering to your user base. You need this Matrix to set a baseline for the volume of user traffic for your analysis:

   

2. Create a Feature Catalog or Taxonomy - a detailed feature breakdown via user actions and attributes. It's a mapping of events, transactions, sessions, clicks, and hovers to the appropriate feature grouping. Check some of the examples here.

3. Create events for product features - depending on how your analytics are set up, you can create custom events or use grouping in Amplitude.

4. Set the baseline for active usage for each feature. Since not all MAUs have equal access to all the features, you should start by setting the expected baseline of users who can access a particular feature.

5. After defining your baselines, it’s important to identify transitional or gateway features. These features naturally have high usage and retention but may not necessarily offer the same value as smaller sub-features. In Amplitude, you can use Journey maps and access_point or entry_point properties to achieve this.

6. Exclude features from your analysis that don’t require proactive user engagement, such as ML-based recommenders, notifications, in-app screen mode (e.g., dark-screen mode), or no-ads experience.

7. Use product metrics that indicate both the frequency and depth of user engagement. I look at what % of total MAUs have access to the feature, then what % of users actually choose to interact with it (AVG DAU), how often (DAU/MAU ratio), how much (Total usage, AVG per user), and how many users are likely to come back (Retention(Day 7), and Retention(Day 30)).

To learn more about how to make sense of product metrics, see examples, and walk through the steps of feature analysis in Amplitude, read a full publication - The Ultimate Guide To Product Feature Analysis.

Big thank you to Paul Levchuk, Stu Kim-Brown, Sonia Wong, Timo Dechau, MyFitnessPal, Strava, and Amplitude teams for their valuable feedback on a draft of this post!

Thanks for reading, everyone. Until next Wednesday!

This month, paid subscribers learned about:

• When To Use Client-Side Or Server-Side Data - The differences between types of data sources and how to figure out which data to use for analysis, modeling, or reporting.

• When To Use Mean Or Median - The cases and examples when it’s acceptable to use Mean vs Median and when it’s okay to use both.   

• An Analysis Of Bias Or Why A/B Testing Fails - A recap of Stanford and Airbnb's collaborative paper on A/B test setup and analysis in two-sided platforms and marketplaces.

  Subscribe now

I can't believe it took me that many months to write an introduction to product analytics.

Last month, I went through a few online classes and tutorials on product analytics on Udemy and GoPractice and checked Reforge, The Product Folks, and Coursera to see what people teach today. I was disappointed that I couldn’t find one class that fits the skills, knowledge, and perspective needed to succeed in product analytics today.

Many classes I saw tend to minimize the entire product analytics domain to only SQL, data collection strategies, or A/B testing. Yes, it's absolutely an important part of it. But there is so much more to product analytics. Knowing SQL and having the ability to extract insight from data doesn’t make you an analyst. Similarly, proficiency in Python does not equate to being a data scientist.

So today, I want to focus exactly on this: What is product analytics? How does it differ from legacy BI, and what skills, qualifications, and experience are required to enter this field? And what does it take to navigate and grow within it?

Why has product analytics suddenly become so popular? 

My take on it is that digital analytics tools like Mixpanel, Amplitude, or Heap have played a significant role in driving this trend. They position themselves with the aim to "understand customers’ end-to-end journeys, improve conversion and activation, increase retention, and deliver a great user experience." The trend gained rapid momentum as more mobile apps were released, and more teams began asking for better behavioral funnels, cohorts, and flow measurements:

The methodology for analyzing user behavior using event data has been around for a long time, but this new era of tools has contributed a lot to the formation of a unique domain and a new type of analyst. 

I'd like to differentiate the domain from the tooling, though. Maintaining the event analytics alone does not make you a product analyst. Many data professionals don’t understand the distinction, often confusing the discipline with tools. Analytics engineering is another similar example.

Skills and requirements for product analysts

Product analytics, foremost, is analytics. Its core foundation remains the same despite better tooling, larger product teams, and a seat at the table to explain what DAU should be.

The essential requirements for product analysts stay the same as any other analyst: 

• Understanding business, strategy, and KPIs

• Statistics

• Modeling

• Causal inference

• Critical thinking

• Data acquisition and collection

• Reporting and dashboarding

Additionally to these qualifications, product analysts should understand how users interact with their products and how they navigate between the screens, web pages, funnels, and banners. What makes them come back, upgrade, share, post, etc. 

🤔 This is an unpopular opinion, but I’m convinced the best product analysts are people transitioning into analytics from UX design. UX designers have a strong intuition of user psychology and perception and understand user journeys well. The better you understand how users interact with your product, the more successful you will be at estimating baselines and seeing the story behind the numbers. 

One of the most common questions I receive is whether analysts should learn SQL and Python to support product analytics. Yes, regardless of the era of no-code tooling, you should be able to extract data, process it, analyze it, visualize it, and make inferences. Personally, I prefer notebooks for analysis over dashboards or Excel reports, so I expect my analysts to be proficient in SQL and Python. However, every company and team varies, and you might be asked to work mainly in PowerBI, Excel, or SPSS (if they are still in the 90s).

📊 Product analytics behind the scenes

Product is a subset of the business. 

As a product analyst, your responsibility involves translating each business KPI into suitable product metrics and then using these metrics to measure product initiatives. For example:

• Monthly retention → into MAU. 

• MRR/ARR  → into successful transactions.

• Churn → into net new cancellations. 

• Subscription renewals → into successful payments.  

It may sound like these are the same things, but they are not. The nature of data is different, and its meaning, lift, and impact are also different.

Product analysts face the challenge of explaining why traditional business KPIs might not be suitable metrics for assessing the impact of product initiatives. These KPIs primarily serve as ecosystem health indicators meant to describe the state of business and safeguard it against various seasonal, external, and micro effects.

However, when conducting an A/B test, relying on metrics like retention might not accurately reflect the lift of an initiative. For example, Variant users might come back to your app 1% more than Control because 62% of them can also be part of other hidden effects you won’t be able to capture or measure, including phone settings, a new TikTok viral campaign, or unrelated factors like plum season or time change. 

Retention is an ecosystem KPI. It is not sensitive to isolated fluctuation or noise. It isn’t designed to be sensitive because it’s an output metric (read Common Mistakes In Defining Metrics by Brian Balfour). However, clicks, page opens, and transactions are good for capturing immediate responses and variations, thus offering a more nuanced view of specific actions or behaviors.

Product analysts should: 

• Understand which metric is appropriate to measure product initiative. 

• Develop a system and calculations to translate changes in sensitive metrics into either a lift or decline in business KPIs. 

There isn’t a universally applicable framework to adopt since every product is unique. My approach involves mapping each KPI to at least 5-7 product metrics and then linking each product metric to 10-15 user actions (depending on what you measure and the nature of your product). This mapping is essential to understand how a 45% increase in signup to paid funnel translates into a 0.002% lift in net new MRR or LTV.

Product analysts work with low-trust data.

We know data is never perfect, never clean, garbage-in-garbage-out, and all of that. But unlike BI and old-school data analytics tied to polished data marts and data lakes, product analysts are also expected to work with very different nature of data - events and their properties and attributes rather than users or transactions.

The difference is that users or transactions come through layers of definitions and processing and are ready to be consumed in a nicely structured format. Every user_id is properly assigned to a user; every translation has a record and ID, and a set of backend services behind responsible for ensuring its trust.

This is not the case at all for product data, which often is event-based (it also may not be). If it is event-based, it is likely to come to your report from the client directly before landing at any backend service or a database that could verify it or put some trust into it (Read more - When To Use Client-Side Or Server-Side Data).

As a product analyst, be prepared for the possibility that up to quite a few of your users could be bots, test profiles, system users, or duplicative users utilizing multiple devices and browsers. Often, confirming the exact number of such users is not easy. Product analysts need the skills to confidently navigate such an environment and develop creative strategies to ensure some degree of acceptable trust in this data. They have to learn to deliver results when numbers don’t add up. And this is where most of us fail.

Ironically, the main purpose of any data analytics team is to represent trust, correctness, and confidence in data reports.

Reporting revenue metrics or business KPIs demands accuracy. So, as expected, legacy BI approaches product data with an expectation of precision and correctness. However, the reality is that there is no precision or accuracy in client sessions, cookies, or events.

Therefore, as a product analyst, your work and routine revolve around establishing checks, validations, and baselines to ensure that your numbers “almost make sense“ or are at least directionally accurate.

Here are some examples of the data health checks I run when onboarding or starting to work with a new product/dataset:

1. Check DAU: Session start vs. Login vs. App open vs. Screen View should be similar and remain consistent across regions and platforms. However, they rarely are.

2. Check if new users are included in the DAU count. Most new apps include new users, but matured products will likely have a method to exclude them.

3. Examine DAU/MAU and WAU/MAU ratios: DAU/MAU is not expected to be over 50% unless “unless.” (It can be above 50%, but that story is different).

4. Check the frequency of usage: If ~30% of users use the app/product more than 2-3 days a week, the WAU/MAU ratio should be above 60%.

5. Verify if the current Retention reporting is based on the same activity metric as DAU. It's often not the case. It should be the same.

6. Check if running ML uses the same activity metric to model recommenders, predictions, and forecasts.

7. Run Day of the Week analysis to confirm weekly cyclicality. What does it look like? When does the highest volume of activity occur? Is it a consistent pattern? What impact do holidays and weekends have? Missing a solid weekly cyclicality pattern is a red flag for data quality (in B2C).

8. Check total views and app downloads against new signups. The number of Views should be higher, but surprisingly, it rarely is. The onboarding success rate (Welcome page to Completed Signup) should reflect the Install to Signup drop.

Many more checks must be done across new traffic, premium, security, retention, finance, and monetization.

Managing context and nuance

Companies heavily invest in product analytics tooling. They expect these tools to provide enough support for stakeholders to self-serve through reporting dashboards and data monitoring. And support experimentation. And hopefully, some predictions. And recommendations.

Today's most popular tooling for product analytics is event-based (remember, even Google sunsetted their 15-year-old analytics to switch from sessions to events). However, events come with an ocean of context that has to be recorded, documented, and simply understood:

Your success as a product analyst will also depend on your ability to create and maintain this context and the nuance it comes with. You have to understand how your product is built and how it generates the data, which analytics are recorded for every app screen, banner, card, upsell, notification, etc. You need to know common app layouts and user flows and understand the difference between funnel analytics vs. loops vs. trees.

I used to joke that as data analysts, we sell our souls to the Data and Platform engineering teams. Any change they purposely or accidentally make can get us either fired or promoted. Well, for product analytics, your best friend should be a Mobile Software Engineer. Developers can design event streams and analytics in different ways. They can configure the system to capture every piece of data movement and send this tracking payload as a massive unstructured bulk (if this is the case, it doesn’t matter what analytics tool you use, as you won’t be able to make sense of your events anyway). Or they can set the logic for it and aggregate it in a particular way, making it easy for analytics tools to ingest and read the data. Know your friends, send them cookies.

As you can see, knowing SQL and Python and having the ability to extract data insights have little to do with succeeding in product analytics. And yet, surprisingly, no one talks about what it takes and what it needs to be a successful product analyst.

To deliver and bring value, you have to learn: 

1. How to set baselines while not having trusted data.

2. How to make recommendations when data doesn’t tell a cohesive story and nothing adds up. 

3. How to support experimentation when you don’t have confidence in the test setup.

For these exact reasons, product analytics is the most exciting and fascinating discipline. It’s the most impactful, and the demand for it is accelerating. If you are trying to decide which domain in analytics to choose, I hope this publication has helped you understand it better.

Thanks for reading, everyone. Until next Wednesday!

Related publications:

• The Frequency Of User Activity: SQL and Analysis

• Growth, Loops, And Some Hard Truths - A Recap Of Amplitude Cohort 2022

• Introduction To Event-Based Analytics

• Google Analytics Termination: Stay Calm And Keep Tracking

• An Analysis Of Bias Or Why A/B Testing Fails

• Do You Over-Report DAU?

• Freemium vs Free Trial Analytics

• How To Locate The Right Frequency Of Push Notifications

• How To Do a Root Cause Analysis

• A Deep Dive into Onboarding Flow Redesign Analysis

• A Deep Dive Into User Engagement Through Tricky Averages

📢📢📢 A quick note! I am on the lookout for 2 analysts-stars: 

• I am hiring a Product Analyst at MyFitnessPal to help us learn user behavior in the app.

• BI Director/AVP to own BI and analytics at SageSure (reporting to VP of Data Science, with whom I had a chance to work before and have only good things to say!)

If you are located in the US and are interested, please apply!

Analyzing why events unfold the way that they do is both an art and a science. 

Today, I want to focus on the “science” part and share an approach and framework for running a root cause analysis to understand unexpected user behavior change, event fluctuation, or metric decline.

I remember being so excited when Amplitude launched its Root Cause Analysis (RCA) a few years ago - the feature that helps you locate anomalies in seconds to understand why events are changing: 

Initially, I was obsessed with it, running RCA against every event fluctuation. And yet, it was not sufficient to answer why something stopped working or failed to perform. The tool pointed us to WHEN and WHERE the decline happens but didn’t help with WHAT and WHY. It helped (and still helps) us to run analysis and react faster, but much work is still needed to answer WHY something is happening.

A framework to analyze WHY the metrics decline 

You would use root cause analysis for KPIs and metrics changes or unexpected user behavior patterns such as:

Read more

Strava is a social network and the leading subscription platform for connected fitness, with over 100 million active community member…

Read more

Subscribe now

Let's say the product team you support is working to re-design the onboarding flow. You are responsible to monitor, analyze, and report on the new flow performance and potential Top of Funnel lift (that there hopefully will be).

Many analysts think that onboarding analysis is easy. 

It used to be easy when it was just on the web. Now with the flexibility of paywall positioning, GDPR screens, and personalized recommendations, onboarding optimizations have become more and more challenging to analyze. 

Today I will share an example of a mobile onboarding flow redesign and will walk you through the process, analysis, and caveats of user onboarding.

A well-optimized onboarding flow is a rare art. It is a thin balance between getting all the necessary data about users, using the smallest number of steps, complying with privacy, and maximizing conversions.

Take advice from famed product growth hobbyists with a grain of salt 

“It's an old myth that you should oversimplify your onboarding to reach high activation rates. In reality, only super low intent customers that would have never activated in the first place drop off (3 screen/9 question onboarding has no more than 10% drop-off rate).” - Elena Verna. 

No, it's not. Data from a lot of case studies points otherwise (here is one - The 8 best user onboarding examples from analyzing 150+ companies, but there is so much data on this out there). Even if we ignore the data, how about having the opportunity to re-engage these “super low intent customers” or Adjacent users?

What is a good onboarding rate?

Onboarding analysis will be different for B2C vs B2B or SaaS. So the benchmarks are, as well.

A successful onboarding flow would capture enough data from users which could later be used for personalization and recommendation algorithms. At the same time, its completion rate ideally stays above 60% in B2B and SaaS and above 50% in B2C.

I’ve noticed investors and growth advisers loop too many things into onboarding, like product activation (which you can’t measure until you have a user in-house) or paying within the first 6 months(and where did “6 months” come from? I have too many questions about the recent free-to-paid piece). 

The onboarding definition I will be using today is related to the steps users take to initially set up their account BEFORE they activate the product or make any purchase. 

Onboarding flow analysis

🍿Getting started

Re-designing user flows is naturally complex because it involves changing user behavior by adding new screens, changing the copy, re-positioning CTAs, and much more. In my experience with heavy redesign procedures, here are some tactics that I recommend:

1. Run a set of smaller A/B tests with one change at a time, rather than one big test with many changes and many variants. You know why. 

2. Start with finalizing the user step sequence first before any copy changes. This is the most challenging step because you don't have a baseline for the new flow. This step includes testing (a) removing screens, (b) adding screens, and (c) changing the order of the screens. Be prepared, as this is the most time-consuming part. You might have many iterations with many multivariable tests. 

3. After the user step sequence is optimized, proceed with CTAs testing. And only then should you change the screen layout. In my experience, re-positioning CTA buttons returns a higher impact than changing the screen layout. 

4. When CTAs are set, continue with the rest of the copy changes, testing (in this order) banners, wording, layout, and then colors. 

5. Once the final optimization is successful, localize (if relevant). Be prepared to repeat the same for different locations depending on your product/service.  

How to decide which screen to start testing with? 

Get onboarding funnel conversion data for every step. Then sort the onboarding screens by each step's conversion rate. Put the highest conversion steps upfront, and the lowest converting steps at the end. Remove steps with the lowest conversions or move them to the very end.

🫕 Pre-launch: 

New screens require new analytics. 

The first step is to work with the product manager and create a catalog of the current onboarding flow. You have to agree on a name for every onboarding step, describe its objective, and document analytics for it.

Once it's done, then your team can begin setting tracking for screen views. My recommendation is not to overwhelm analytics with events and to set up only the main event for every screen, e.g. screen_view. 

In some cases, it's also helpful to create: 

1. cta_click (IF there is upsell, survey, or else important. I'd ignore clicks to "next” screens)

2. secondary_button_click (if relevant) or skip_screen event for the optional screens

But frankly, in 99% of cases of modern onboarding flow, I'd ignore these two.

🚀 Launching tests

1. Hypothesis: you will have different hypotheses for every test. For example, you may start with "Removing the email confirmation step will increase the number of successfully completed signups". Or "Moving location_screen before preferences_screen will increase the number of users landing on preferences_screen" and so on. 

2. MDE: for the user flow testing, all your MDEs are likely to stay small. Your Top of Funnel traffic, I assume, gives you a large sample size. So even a small lift in conversions will be impactful. (It’s a reason why I like onboarding tests. You might change one little thing and its lift will echo in every downstream metric).

3. Test timeline: it’s a good thing Top of Funnel tests usually run fast. Be prepared to run multiple tests in a sequence. I remember it took us over 4 months to finish more than 15 tests.

4. A/A: given you introduce new events, and will be testing new user flow without historical data to compare it against, it's recommended to set between 2 to 4 days for the A/A test to make sure the experimental platform works as expected and users are randomized correctly. Once it's done, you also can use this data to confirm the timeline for the test to reach significance. Win-win.

My recommendation is usually to launch treatment to 5% traffic for a few days to monitor the initial data. Especially for the first round of user step sequence iterations.

If you have confidence that the test is unlikely to make a negative impact (removing the Email confirmation screen, moving the How did you hear about us screen to the end, eliminating other screens that requested more user data), you can skip the slow rollout and launch a test to the larger traffic. I still suggest going slow with the new screens to make sure you won't lose many users if things go wrong.

💔 Too many screens, too many steps

The most impactful onboarding redesign project that I remember ran for over 4 months. Our initial S-T (Signup to Trial) was 0.08%, and after further analysis, we learned that only 25% of users successfully made it through the onboarding flow. So we were losing 75% users in the flow which we paid to acquire. Harsh.

The initial onboarding funnel included 9 screens, starting with a common Welcome screen and ending with the Upsell. It also included an email verification step. Every further screen was capturing some type of user data - demographics, location, and preferences. Mind you, we didn't even have to worry about the GDPR screen at that point. And yet, after confirming that only 25% of users made it through the flow, it was clear that we had to re-design and optimize it.

📊 Our analysis and learnings: 

Every test was performed differently, as expected for the onboarding revamp. We had clear winners and some losers, and we also had a few inconclusive tests which we launched again with another flow or copy iteration. As a result of multiple multivariate tests, we optimized the onboarding flow by 220%! After the redesign, 80% of users successfully completed all onboarding steps (compared to 25% with the old flow).

💡 What we learned:

1. The biggest step to cut off users was an email verification wall. Once we removed it, we jumped up to 50% more successful signups. 

2. The marketing survey (how did you hear about us) didn’t seem like a big deal but was costing us 30% of users. We removed it from the onboarding flow (another marketing initiative was launched later to gather this data for already registered users). 

3. The location screen that was requesting the user address was removed from the flow, along with a demographics step. We made these optional screens and encouraged users to fill them out on their 2nd visit. It increased completed signups by 60%. 

4. Another win was to remove the product tour video from the onboarding steps and offer it after the registration was completed. It gave us another 40% boost. 

5. Most of the newly tested copies with round buttons and dark screens boosted conversions by at least 15%. 

6. We ended by putting the highest-performing screens at the beginning of the funnel. The lowest-performing screens were removed. If we had to keep some, then we put them at the very end of the flow. Shifting scenes around improved the funnel completion by almost 50%.

7. Changing the positioning of the upsell screen was the most impactful for monetization. To my surprise, the highest trial conversion was received after testing the upsell screen as THE FIRST screen users see in the onboarding flow - before the welcome screen. As I also learned recently (and which confirms my findings) is that apparently, users are the most likely to convert in the early steps. Right after the Install event. The further down upsell is pushed, the lower conversions it returns. I find that fascinating.

8. A general rule for me is a 40% improvement in onboarding conversion returns up to 4% S-T (Signup to Trial) and 3% I-T (Install to Trial) improvement. Optimized onboarding flow can return up to a 1% to 2% increase in DAU and Retention Day 1. It is unlikely to impact MAU, Retention Day 7, or Day 30.

9. Testing the same onboarding changes on different platforms may show completely different results. 

10. Overall, it takes about 4-6 months to fully re-design the flow (depending on the current stack and analytics).

Summarise: 

1. Reiterating onboarding flows is one of the most impactful product initiatives to move Top of Funnel conversions.

2. Onboarding flows can be difficult to measure. Most modern flows are not funnels but trees, or worse - reverse trees. Depending on the user's answer, the flow might reverse and change the sequence of the steps. For analytics, it implies you might need to replicate over 10 different multi-step funnels and build reporting for every variation of user flow. It can be a lot. 

3. As a compromise, I have seen many teams agree to report only high-level funnel, e.g. welcome_sreen to signup_completed. The downside of this approach, when you test multiple iterations of the flow, you won't be able to use this high-level Baseline as a success metric. And in many cases, analytics for the interim screens are missing.

4. Keep event analytics light. Don't create a new event for "next", "back", "learn more" or other CTAs. Stick to the screen name events only. You should be able to build funnels just by using high-level screen analytics.  

5. Use slow rollouts and A/A tests, especially if you add new screens. 

6. Onboarding analytics quickly becomes messy unless you organize and maintain it. Start with the event catalog to record screens and user step(s). Update it with every flow iteration. It does take a lot of work. If ignored, your analytics team won't be able to report on which onboarding steps underperform.

7. Start with finalizing the user sequence first. Then CTAs. Then Copy changes. if something does not work out, it's easier to reiterate and change the Copy rather than the steps sequence flow.

8. Put the highest conversion steps up front. Remove the lowest converting steps or keep them at the end.

It's proven that the shorter your onboarding flow, the better it performs. I see more apps go for not having onboarding at all, and put the upsell screen as the first screen users see after the welcome screen, or activation CTA as the first initial step. Users have the highest motive when they install your app. Don't waste it on onboarding.

Thanks for reading, everyone. Until next Wednesday!

Related publications:

• Why People Don’t Buy Your Subscriptions

• A Deep Dive Into User Engagement Through Tricky Averages

• How To Prove Causation

• Embracing the New Era of Accelerated Testing

• How To Get Average Logins Per User Per Day

• Do You Over-Report DAU?

• How To Measure Your Adjacent Users

• The Ultimate Guide On Churn Reporting (And Its Technicalities)

• How To Locate The Right Frequency Of Push Notifications

• Introduction To Event-Based Analytics

• Playbook For Launching, Monitoring, and Analyzing A/B Tests

Subscribe now

If I told you averages are tricky, you probably know it already. We all know how outliers, variance, and distributions create a misleading story. That being said, as a proxy for 'typical', averages will not disappear from reporting, neither in finance, nor marketing, nor product. 

I am here today to remind you of the tricky (very tricky) nature of averages. I’ve been burned by it so many times before, and just recently it happened again (you would think someone learns from past mistakes). I will also demonstrate an example of 3 different approaches to report the same average metric to measure user engagement - which each time leads to a completely different story.

As always, I’m here to remind you how fascinating analytics is, and how we have the power to re-create any story and tailor any stats VCs might want to see. And we should use this skill wisely.  

Averages are my pet peeve. I don’t believe it’s safe to read too much into KPIs that are calculated on averages. For example, I shared earlier my concern with ARPC (or ARPU): 

Average Revenue Per Customer will rise when you are losing customers. Just think for a moment about this. It’s not only ARPC. When the total audience shrinks, every metric that incorporates “average” and “per user” will increase. Thanks, math!

Below I’ll demonstrate an example of how averages can mislead, trick, or create a completely different story.

A case study on user engagement analysis

Read more