This Is Why You Overcount Your Daily Active Users - Issue 88
This Is Why You Overcount Your Daily Active Users - Issue 88
A follow-up from the Engagement and Retention series. Why measuring app activity can be tricky, and how to best capture daily user activity events to report DAU and MAU
Hello analysts, I am back to entertain you with another riveting story about how challenging and tricky Active User reporting is.
Many years ago when I started my analytics journey, the world was quite simple. Once upon a time, there was a website where users logged some activity. Analysts would pick their recurring Login or View event and make it a DAU metric. Nice and simple. We pulled these daily activity events into some materialized views and used them as a source of truth for daily, weekly, and monthly engagement reporting. We could also JOIN the same view to whatever experiment data we could pass and report on DAU impact for a given A/B test. Piece of cake.
But the world evolved. Given that too many Product Owners by nature have unrealistic ambitions mixed with limited understanding of execution, the new shit storm age of Customer Data Platforms arrived, making analysts' lives painful and sad. We would lock all user data into one hell of a “single view of the customer” or whatever the selling pitch was, losing user traces on the way here and there. Analysts lost flexibility and ownership of the data ecosystem, and every simple data request became a multiple-week project requiring ETL changes.
Buy a Single Customer View dream for only up to 300K annually
Then the era of mobile apps came in. Every respectful website needed an app now. With years passing by, analysts would notice how mobile activity would rise and often outrun website usage. I remember being astonished watching how multiple onboarding conversions were twice as high on mobile at different companies just a few months after the mobile app was released. Later, Apple Pay and Google Pay would meet or even boost payment conversions higher than desktop Stripe and Paypal.
Then the famous problem of merging and mapping mobile and website data arose. Every time when Product Owners heard “user journey” or “user lifecycle data” they bought All The Things. This is where Segment was supposed to help. It worked effectively less than 1/3 of the time, but damn it sold well. Around the same time, marketing attribution became a thing, and every CEO would deliriously exclaim “Why is the “unknown” source so high?! I need to know where all of these users came from!!” And they would spend a lot of money, effort, and many months on fixing it only to get 2-3% more trackable ids.
It all went downhill from there for analysts. A never-ending battle to chase attribution to optimize spending on paid marketing. Chasing attribution to get cheaper users with a higher probability of conversion is still a questionable road to success, in my opinion. But prove me wrong.
The reason for this trip down memory lane is to point out that all of these changes led to a different view and approach to measuring and reporting user activity. Overcounting users (either signups, activity, or even paid subscriptions) became a common and expected pattern now. Today, when we say MAU, it is not the same definition and meaning for the same company as it was 5 years ago. A few years back, a 2% increase in DAU would be a victory to celebrate and likely a result of a new feature release or product optimization. Today, a 2% DAU increase likely means a background app event tracking issue or a bug with the session start event.
A few weeks ago I published Engagement and Retention, Part 2: Daily Active Users where I walked through how to define active users, what events to use, which activity frequencies to follow, and how to approach engagement analytics for weekly and monthly reporting.
To recap, here are the most common events used for the “active” user definition across many companies:
Visit
Session starts
Login / app open
Web page view/screen view
The main user activity (item view, search, exercise log, transaction, etc).
I went through these in more detail and explained that by using app open or session starts, you are likely significantly overcounting user activity.
If you are using DAU activity based on Amplitude Session Start events, you are likely to overcount your DAU as well. In this article iOS Silent Notification Problem — How they mess up your app's lifecycle and Amplitude session events the author goes through SDK Amplitude integration explaining how the silent notification system wakes up your app in the background and initiates sending analytic events and attributes, like Session Start. It means you will count activity for users who are not actively engaging or opening the app at all. Interestingly, Amplitude itself encourages not using these events for DAU reporting. Please read their You’re Measuring Daily Active Users Wrong piece. The author is a little confused between engagement with retention reporting, but overall her approach to DAU events is good.
As I had to look at the Session starts overcounting issue last week, I want to reiterate how vital it is to build your DAU KPI based on the main user activity that illustrates the core app usage. I’d ignore all the “noise” and vanity events like visits, app_opens, session starts, and screen views, and tailor KPI to only the main transaction or event that you expect a user to most often do.
Thanks for reading, everyone. Until next Wednesday!