Do You Over-Report DAU? - Issue 139
Do You Over-Report DAU? - Issue 139
Yes if you use Sessions. Why measuring app activity is tricky, and how to best capture user engagement to report accurate DAU and MAU
My good friend (who is a Head of Analytics at a popular eCommerce app) recently shared a gripping story about how they overcounted DAU in a digital analytics tool for a few years by at least 30% using session events.
I decided to share their story today because (a) the issue with over-reporting user sessions is very common (I actually wrote about it a year ago - This Is Why You Overcount Your Daily Active Users. And then a half year ago again - Growth, Loops, And Some Hard Truths) and (b) its impact can be substantially negative. DAU baseline is a foundation for many PLG metrics and KPIs, including retention. If you over-count it, every downstream engagement metric becomes skewed. And (c) there simply isn’t much info on this subject out there. I am surprised to be one of the few sounding an alarm to the world about the danger of session events for analytics.
In this publication you will learn:
Why you might be likely to over-report DAU and MAU for mobile apps.
Common pitfalls with user activity events.
Session event differences across popular applications that report user engagement analytics.
My recommendations on how to set up proper analytics for DAU reporting.
Before we dive in, a quick recap on the mobile analytics apocalypse.
How the error was introduced
The story is as old as time. iOS or Android developer implements a typical SDK integration with an analytics tool (e.g. Mixpanel, Amplitude, Braze, Google Analytics, etc). The new events pass Q&A, and the product manager begins using whatever is available for measuring user activity. In 90% of cases, these are start session, app open, or screen view events.
As the team grows, the first analyst comes in and adopts the event with its expected behavior and accelerates the reporting. They may or may not run their own due diligence for that event - making sure it's in line with other activity events, and that the volume looks right.
The Start Session event indeed was firing as expected - when the app was brought into the foreground and a user launched a session. It truly recorded every user who activated an app. Unfortunately, not only users can do that. The system may wake your app in the background via notifications, certain types of transactions, securities, etc, and send analytics to the digital platform you use. So the Session Start event was firing for users who didn’t open the app. In other words, they were still counting activity for users who were not actively engaging or opening the app at all. (I bet they hit the DAU benchmark though!).
And because this was the case since their analytics was initially set up, there was no red flag with this event and no easy way for analysts or PMs to notice the over-count.
And it’s not only my friend and their app who was impacted. I brought up a similar problem last year by sharing the case study from 7apps - iOS Silent Notification Problem — How they mess up your app's lifecycle and Amplitude session events. A similar story with the same impact - the author goes through SDK Amplitude integration, explaining how the silent notification system wakes up your app in the background and initiates sending analytic events and attributes.
I believe many apps that use the Session Start event for DAU are affected by this issue. If you do, I am sorry to break the bad news to you. But you count non-active users in your DAU and MAU.
Common pitfalls with user activity events
Here are the most common events used for the “active” user definition across many companies: visit, session starts, login, app open, web page view or screen view, user click, or other activity (item view, search, exercise log, transaction, etc).
Each of these events has its own downsides and limitations to be used for DAU reporting:
Visit
The web event is used for DAU quite rarely (in my experience). It will count unauthenticated and not signed users as active unless you filter them out. This is different from webpage view and often stands for anyone (and anything including bots and fraud users) landing on any webpage. Visits are the most unreliable, not clean events, and are the reason why I don’t work with marketing analytics 🚫.
Session Start
Mobile and web events, which makes it so appealing to use for reporting if you support multiple platforms. They also include unauthenticated IDs and not signed-up users unless you filter them out. The challenge with Session events is that they are defined slightly differently in every analytics tool:
Google Analytics - A session is a period of time during which a user interacts with your website or app. By default, a session ends (times out) after 30 minutes of user inactivity. There is no limit to how long a session can last.
Mixpanel - The app session executes when the user spends more than 10 seconds in the app and with no limit on the maximum session length.
Braze - A session is started when the application is launched and the app enters the foreground and ends when the app leaves the foreground or when the app dies. The default value for the session is 10 seconds.
Amplitude - A session is the period of time a user has your app in the foreground or has your website open. For mobile, a session begins when the app is brought into the foreground; it ends when the app goes into the background and no events are fired for at least 5 minutes. Web sessions time out after 30 minutes by default.
The definition sounds similar, but 4 different tools will give you 4 different values for a session (and thus, 4 different DAU metrics).
Understanding the flow of sessions can get quite complex. I really like Mixpanel documentation on sessions and their examples of tracking differences:
A single user can start multiple sessions. You need to work with your developers to understand how events and sessions are connected and defined. Also, depending on the user's device, browser, and privacy change mode, they can be treated as different users. I believe this is the most common issue with DAU over-count in digital analytics tools.
All in all, sessions are tricky and complex. My recommendation is to avoid using session events for KPIs or for anything important.
App Open
The most common mobile event. It stands for activating or opening the app. With app opens you can do both - over-count and under-count users.
Some analytics tools differentiate the first_app_opened event from app_opened. In this case, your app open is a consecutive event after the initial first app open, and it should be clean. If the first_app_open is not available, you will need a way to exclude new users with their first app open event. Otherwise, you will count new users in DAU and MAU (unless it’s intended).
Web page view/screen view
The challenge with page/screen views is that you must have analytics set up for EVERY screen your app or website has. And every time you add a new screen, you have to make sure you add new analytics for it with the same property and event naming you already have. If your DAU is a custom metric built on top of the screen view event, you will need a process in place on how to edit and refresh the metric logic with every new screen release.
User action
If you read my older publications, you know that I highly recommend and can’t emphasize enough how vital it is to build your DAU and MAU KPIs based on the main user activity that illustrates the core app usage:
I’d ignore all the “noise” events like visits, app_opens, session starts, or screen views, and tailor KPI reporting to only the main transaction or event.
💡Read an interesting take from Patrick Campbell about passive anti-active usage products where he highlights the importance of measuring the value users get from your product rather than tracking user activity. The way I understand it is that the volume of logins, app opens or sessions don’t correspond with the outcome or value you want to measure to better understand user engagement. Don’t get fixated on tracking noisy activity.
Overcounting users (either signups, activity, or even paid subscriptions) has become a common and expected pattern now. A few years back, a 2% increase in DAU would be a victory to celebrate and likely a result of a new feature release or product optimization. Today, it likely means a background app event tracking issue or a bug with the analytics event.
App development never stops. With every release, new events are added for every view, click, toggle, hoover, etc. Event analytics becomes overloaded. Many custom metrics and conversions are created on top of existing events, and it’s easy to get lost in which events are primary, foundational, or simply noise. Most team resources are stretched thin towards setting up and expanding new analytics and reporting. It’s very rare that known and trusted events are being Q&Aed again (unless their volume changes).
Teams tend to trust whatever is in their product analytics tool. We assume events mean what they are supposed to mean:
active users stand for users who actually use the app
the app open event fires when a user is opening an app
or the start session is recorded when a user is activating an app.
But that’s not always the case, and there can be myriad variations. That’s why setting up a solid product analytics practice at your company is so important.
I’ll end with
newsletter quoting very smart things I say 🤩:
Thanks for reading, everyone! Until next Wednesday!