Inside Product Analytics: Decoding User Behavior Part 2 - Issue 278
Deep dive into product analytics: required skills, tools, projects, and navigating challenges.
Welcome to the Data Analytics Journal, where I write about data science and analytics.
This month, paid subscribers learned about:
Upgrades and Downgrades: Where Most Reporting Goes Wrong - How to measure and report customer upgrades and downgrades, how to make winback reporting accurate, and how to segment all types of returners.
Rethinking A/B Testing for B2B and SaaS - Why A/B testing for SaaS and B2B is different from experimentation in B2C. Lessons from StatSig on best practices for designing and running experiments in B2B.
Analysis for Optimal Cadence and Frequency - Applying data science to marketing analytics: how to identify the right frequency for upsells, ad impressions, or notifications.
Before we dive in, a quick announcement:
On September 25, StatSig is hosting the StatSig Significance Summit - a conference for product builders in San Francisco, where product and data scientists come together to share their learnings. Speakers include experts from Figma, Grammarly, Atlassian, Anthropic, Lift, Linear, Notion, and more.
I’m partnering with StatSig this year and excited to give away a few free tickets to my readers for the in-person event ($250 value). Email me if you would like to attend!
Back to product analytics.
My first introduction to product analytics was published almost 2 years ago, when I covered why the field emerged, why demand for it remains high, and what skills and qualifications are needed to enter product analytics.
Today, I’m taking it a step further and diving into the projects, tools, and frameworks we use in product analytics - how to break down the user lifecycle, how to approach analytics for different products, and what the market landscape of product analytics tools looks like.
Intro to product analytics - a quick recap
Start here - Inside Product Analytics: Decoding User Behavior
User behavior tracking tools have been around for years, but modern event-based approaches have shaped a distinct domain - and with it, a new type of analyst. An analyst who understands how users interact with the product, how they move across screens, pages, funnels, and features, and what drives them to return or upgrade.
Today’s product analytics tools are event-driven, but events only matter with context. Your success as an analyst depends on creating and maintaining that context - knowing how the product generates data, what’s tracked on each screen or feature, and how to interpret flows, funnels, loops, and trees.
Being a product analyst is not just about managing event data. Product analytics is still analytics at its core. You need the same fundamentals: understanding business strategy and metrics, statistics, critical thinking, data collection, reporting, and dashboards.
You need to know which metrics best measure a product initiative, and how to translate shifts in sensitive metrics into real changes in business KPIs. A product is a subset of the business. So our day-to-day work is about establishing checks, validations, and baselines to ensure numbers are directionally accurate and tell a story.
Product analytics is centered around the product.
Unlike marketing, business, or finance analytics, product analytics is grounded directly in the product itself. Product analysts are fully immersed in understanding how the product works.
A product can be an app, a website, a browser extension, a hardware device, a feature, a piece of software, or a model - essentially any type of application. The better you understand how the product is built, the more effective you’ll be as an analyst.
Easy to say, but in practice, this means there’s a steep learning curve. Transitioning from supporting a SaaS feature to owning onboarding for an Android app is not something you can get up to speed on overnight.
For example, if you support a mobile app, you need to know whether it’s native or cross-platform, which SDKs it uses, how data is collected, how StoreKit works, and how tracking is implemented. And it doesn’t stop there. You must also understand how the app stores work, the technical differences between Apple’s App Store and Google Play, and the full lifecycle of an app version. Mobile stores come with their own tools and metrics. You need to know the types of monetization products offered:
There’s a big learning curve with mobile app analytics.
But one of the most difficult products I’ve ever supported was VidIQ, a browser extension and platform for YouTube growth. Extensions are very different. If you have ever worked with extensions, you must know - users resist adopting them at first, but once they do, they often stick with them for a long time. The user dynamics in an extension are very different from web or mobile apps. There are no standard user flows or navigation patterns to borrow. Retention, onboarding, and usage don’t fit into any benchmarks or proven frameworks. Every A/B test or feature adoption behaved differently on the extension compared to the app. That was frustrating. We couldn’t apply the same learnings or assumptions.
Anyway, every product type, whether a mobile app, web app, or extension, brings its own unique challenges. Since analytics success depends directly on how the infrastructure is set up, as a product analyst, you need to understand how the product is built, how tracking is implemented, and how analytics is generated.
How to get there:
Learn to read documentation. Most analysts chase speed and skip onboarding walkthroughs. We don’t take the time to go through “getting started” guides or README instructions. But we should.
Be curious and active in the community. Every open-source tool comes with a Slack group or forum. Most tools host office hours, webinars, and publish newsletters. Use the help desk, talk to support teams, ask questions, and seek help.
Attend events or watch recordings. There’s a reason I cover events, share free tickets, and write recaps in my newsletter. For example, I wouldn’t know that Apple now offers winback offers (driving up to 20% more paid subscriptions from resurrected users) if I hadn’t watched WWDC. Or that for SaaS A/B tests, you must randomize at the organization level, not the user level, if I hadn’t attended Data Council.
These are random examples, but the point is: yes, it takes effort to read, watch, and filter through the noise. You’ll only use a fraction of what you learn. Still, being a few steps ahead of your peers does make a big difference.
The product is centered around the user.
Let’s talk about the kinds of questions and projects that shape product analytics.
Any type of product (an app, an extension, a feature) is designed for a user. That user might be a human, an agent, a model, or even another piece of software. A user can also be a customer, a consumer, or a seller.
That’s why I approach product analytics from two angles: user lifecycle and personas. Most projects I work on start with the user lifecycle, which would be broken down for every product persona.
User lifecycle
Most of our projects in product analytics are centered around a specific stage of the user journey. For example:
Signup → Onboarding → Activation → Monetization → Retention → Winback
Of course, this lifecycle looks different depending on the product:
Mobile apps don’t really have “signups.” The journey starts with a download. A signup might represent the completion of onboarding steps, or it could happen before or during onboarding.
Browser extensions or products with long or complex integration flows often define activation around a successful installation or completed integration.
Subscription products and SaaS often simplify retention to the inverse of churn: if customers didn’t churn, they’re considered retained.
And so on.
A product analyst’s responsibility is to set up measurements for each stage of the user lifecycle, translate business KPI into suitable product metrics, and then use those metrics to evaluate product initiatives. For example:
Monthly retention → into MAU.
MRR/ARR → into successful transactions.
Churn → into net new cancellations.
Subscription renewals → into successful payments.
Examples of user lifecycle analyses
Over the last 2 years, I’ve shared many examples and my frameworks for each stage of the lifecycle:
Signup and Onboarding:
Activation and early engagement:
Monetization:
Retention:
Winback
Understanding user actions remains a challenge
To be clear, the challenge is not capturing user actions, but reading how these actions come together to form a behavior. And then, understanding what that behavior actually means.
There is no shortage of product analytics tools
But there is a shortage of analysts experienced with these tools
Some tools are better for smaller companies, others for enterprises. Some offer freemium plans or are open source. The most successful product analysts are familiar with many of these tools and can advise product teams on which tool is best suited for a given user volume, revenue milestone, and team plan. Product analysts should know which of these tools are event-based or session-based, whether they sit on top of client-side or server-side data, and which approach is more suited for a team and their reporting.
I plan to do a deep dive into each of these tools over time, though it may take a while to cover them all. I’ve also noticed a few are missing from the map, like LiveSession and Plausible.
What it takes to be a successful product analyst.
The job market is full of analysts proficient in SQL and Python. There are fewer who are grounded in statistics, and when they are, they often move toward ML engineering or NLP rather than analytics itself (very unfortunate). Even fewer have hands-on experience with modern product analytics tools, know the best practices inside and out, and can deliver quickly.
To deliver and bring value, you have to learn:
How the product is built, how tracking is implemented, and how analytics is generated.
Who the user is, what their lifecycle looks like, how teams expect them to behave, and how they actually do.
What the product analytics tool landscape looks like: who the leaders are, who challenge them with bold new features and vision, and which tools are more niche but better suited for specific domains.
I keep saying it: product analytics is the most exciting and fascinating discipline. It’s impactful, and demand for it is accelerating. We need stronger product analysts, we need them now - and I hope my newsletter inspires you (or will inspire you) to become one.
Thanks for reading, everyone!