Why Your Activation Analysis Is Wrong - And How to Fix It - Issue 247

How to locate and validate Activation using product analytics tools

Welcome to the Data Analysis Journal, a weekly newsletter about data science and analytics.

---

A year ago, I shared my analysis on Activation I did at MyFitnessPal. Many apps reached out with questions about modeling, performing regression in product analytics tools, and handling tricky Activation cases.

So today, I want to follow up on Activation and show you how to identify a-ha moment, different ways to calculate it, and what to do when you’re not getting a clear signal or your data is inconclusive. I’ll also walk you through how to get Activation in product analytics tools and validate it through analysis.

If you’ve been reading any PLG-related content, you’re likely familiar with Activation or a-ha moment. I believe it initially gained popularity with Facebook’s famous “7 friends in 10 days” and was later revived with the PLG movement.

Today, I won’t be discussing the concept of Activation but rather how to calculate it using product analytics tools or modeling. If you are new to Activation, please start here:

1. Activation: The Product Metric Everyone Thinks They Need But Can’t Seem To Define

2. Rows: How to pick your activation metric

3. Crafting The First Mile Of Product

4. What is a good activation rate

Locating Activation milestone in product analytics tools:

Here’s what 90% of growth marketers do to locate Activation:

1. Create an onboarding funnel that tracks every onboarding step.

2. After the last step in the funnel, add one more step for the first successful key action users complete (e.g., create a report, listen to a song, log an exercise, etc.) and call it the Activation rate.

3. Set the funnel to 1, 4, or 7 days.

4. Done.

This is an incomplete and inaccurate activation funnel. However, it’s the best you can do using the Funnels feature in product analytics tools.

Then, analysts step in and create a cohort of returning users who completed the key action after returning within X days. We do this using Retention or Heatmap features - not the Funnels.

But this still isn’t Activation. It’s simply the % users performing the core action on a given day. It might indicate Activation - or it might not. And there is no way to definitively know unless you leverage Compass in Amplitude or Signal in Mixpanel. However, both come with data context and caveats. Let’s break it down.

Calculating Activation - Expectation vs. Reality

Expectation:

You open Amplitude and instantly see your Activation milestone clearly within seconds:

Reality:

Such analysis worked for MyFitnessPal because (a) I spent months setting up and fine-tuning analytics there - we had to reset events, remove noise, configure retention definition, and more, and (b) we had 10 years of data.

If your app was launched recently and your analytics weren’t set right, your Activation report is likely to be inconclusive.

It doesn’t mean your product doesn’t have an Activation moment. It just means you’re not capturing it correctly. The missing signal could be due to noise in event tracking, outliers skewing data, insufficient data, skewed distribution, or other data inconsistencies.

If this happens, follow the steps below to manually find your Activation milestone using product analytics tools. Whenever I feel lost in data or struggle to capture the signal, I use this framework - and it works every time.

How to do Activation analysis

Step 1: Start with the most trusted event for a product feature

Begin with a clear, well-defined “action” event that signals when the activity is completed, such as transaction_completed, food_logged, report_submitted, etc. Ignore less meaningful events like views, clicks, toggles, banner views, etc. These often introduce noise rather than actual user engagement.

Step 2: Create cohorts based on feature usage.

Define cohorts based on specific feature interactions, such as Users who logged foods at least once, Users who logged exercise at least once, Users who integrated a new device at least once, etc.

Each persona should have its own cohort.

For a typical app, you’ll likely create 10-12 cohorts. Each cohort must be modeled separately. This is a lot of work. It won’t be done in a day.

Step 3: Create 2 reports for every cohort:

For each cohort, generate 2 key reports:

1. How many times users completed this particular action (once, twice, 5 times, 10 times, etc.)

2. When they completed this particular action (Day 0, Day 1, Day 2, etc.)

Step 4: Define the retention metric for Activation.

I don’t use Retention based on built-in Amplitude or Mixpanel values like New User, Any Event, or Any Active Event. If possible, my retention is narrowed down to a very particular returning action, like reading a book, logging food, submitting a request, or completing a transaction, etc.

However, for some apps, a simple app open or screen view event may work just fine. The key is to select a high-level event that users can do on every return but is still well-defined and meaningful.

Step 5: Run correlation analysis

For each of the reports for every cohort, run a correlation analysis comparing feature usage with multiple retention timeframes: Day 1, Day 7, Day 30, Month 3, Month 6, and Month 12. That’s a lot of plots like these:

This is where most people stop.

This is NOT Activation yet :). But we are getting closer.

Step 6: Run regression modeling for the key 2-3 patterns

I described the steps and process here - How to do linear regression and correlation analysis.

What’s important to understand is that Activation is predictive of user retention. Whenever you see “predictive,” it means you need to either model it or run an A/B test.

For modeling, it doesn’t need to be complex ML. In the example above, I used linear regression, and it worked well for me. Linear regression takes correlation analysis further and shows how much one variable affects another and whether you can use the pattern of one variable to predict and estimate the behavior of another.

Linear regression doesn’t prove causation, but it does give you more confidence that there’s a strong connection between variables and how this connection is expected to change if you increase or decrease variables.

For this step, you can use this quick template - Correlation and Linear regression template. Run regression analysis on the 2-3 strongest patterns from your correlation step. Read Decoding Regression Scores to understand how to interpret regression output. The regression model with the highest R-square value is your Activation milestone. Congratulations - you did it!

What if it still doesn’t look right?

Sometimes, even when you follow best practices, the data doesn’t return the expected output. Or it doesn’t make sense.

• Check if your data is significant. This is the most common cause of flawed analysis. You may not have enough data for certain actions, even if they are core features. If this is the case, you have to work with qualitative data - use user surveys and research. You can’t model Activation just yet if the data isn’t there.

• For subscription apps, measure Activation against paid renewals. Do NOT measure Activation against just any retention metric in Mixpanel. Activation should be tied to paid renewals for each plan you offer. If your app is new, you may not have enough renewal data, especially for annual subscriptions. In this case, you may try your luck with a proxy metric - use activity data for paid users.

Thanks for reading, everyone. Until Wednesday!