Analytical Case Study On User Engagement - Issue 68
Analytical Case Study On User Engagement - Issue 68
The methodology of a common case study for data and product analysts on user activity and engagement
Let’s say you are a data analyst at Twitter, and your team noticed the average number of tweets has dropped 20% in 2021. What analysis would you do?
Investigating metric changes is my favorite part of being an analyst. Sometimes it’s a sudden and significant decrease that requires you to drop everything and get a compass and a magnifying glass. An analyst’s life is never boring. I don’t work at Twitter, so I don’t know if such a 20% drop is expected variance or out of the ordinary. For this analytical study, I’ll assume that it’s not expected, and we have to conduct our analysis to explain it.
When analyzing any metric changes, first things first: you work on ensuring the data you see is correct. Find at least two or more other data sources showing a similar decrease to confirm it’s true. Most likely, every source you look at will show you something else with a different pattern, making this whole thing a suspenseful hunt for clues. But let’s assume that you confirmed the 20% decrease of average tweets per user over the last year for this hypothetical case. Now things are getting interesting!
From Medium: Fooled by stickiness
📊 Analysis
The metric to measure tweets is probably the average number of tweets per DAU. Our task is to explain why it dropped 20% starting this year.
We begin with generating multiple hypotheses that have to be either confirmed or rejected. Before we start digging into the data, let’s group our hypotheses into three main groups: product, market, and user.
1. Product
To confirm if the drop is related to a product bug or a specific product launch, we should look at differences across different platforms (mobile/web) and devices. The product-related cause would only affect a proportion of users who are using a certain product. Check all the product launches around the time when the drop becomes noticeable. In this case, your drop is likely to be sharp.
You will need to create a time series chart of tweets per user broken down by platform or device for this hypothesis. Ideally, show another line of users who use different features to illustrate the issue with a specific product launch.
2. Market
If the decline we notice is not related to the product itself, our next hypothesis is market (or competition). To confirm it, we would need to see a gradual decline rather than a sharp drop. It would also affect other activity metrics, like views, retweets, comments, etc. Check country and user location and region data. Are any specific market groups affected, like teens, for example? Or gamers? Or other specific business types?
For the Market hypothesis, the best visualization is a time series chart of tweets per user segmented by country or age (or any other demographics you are able to identify) along with other activity metrics showing a similar decline. If all engagement KPIs are affected by one market group, that could be a competition cause.
3. User
If none of the above worked for you to confirm the drop, we move to the most exciting part of our analysis - user. Overall, if you are stuck with any engagement analysis or direction, always follow a user. It will lead you to the light. I break a User hypothesis down into 2 sections: user growth and user behavior.
User growth speaks of a user base - essentially, how many users use the product. Work on checking the drop against different user cohorts - registered but not active, active, but lapsing, super active (power users), premium or free, business or individuals, or whatever personas you work with. If user growth is the reason for a decline, you will see that only a specific user cohort is affected. To demonstrate it, you will need to create a time series chart of tweets per user per visit or a session, split by user persona or user type (business/individual). You should have some not-affected personas in your chart to prove that only some user groups are affected.
User behavior is similar to user growth, but instead of user types, you dig into user actions. Your goal is to show that only some pattern of user behavior is affected. For example, you may notice that along with average tweets per user, average comments per user and average likes per user are also declining, but overall sessions, visits, or time spent on an app don’t change. If that is the case, this is a larger issue you now have on hand which you would need to brainstorm about with different stakeholders and partners. It may lead to users “outgrowing” or “maturing” your product and might be suggesting that it’s time to rethink the product offering. The best way to represent user behavior change is a chart showing different user actions (tweets, likes, retweets, messages, etc.) compared to a time series of overall time spent on an app (apparently I like time-series charts). You have to show which user actions are declining and which don’t change.
Another hypothesis I haven’t mentioned yet is content. This is relevant for Twitter or any other social media that offers different types of content on its platform - private, public, ads, promotions. It’s quite possible the 20% drop is happening for only a certain type of content. To prove it, you need to show a chart with a proportion of average tweets per content and segment it by private/public/ads and show this proportion before and after the change.
Consider also external factors like a pandemic, war, or big social movements (#MeToo, BLM, etc). These may cause a sharp decline (or an increase) in usage across many top user actions, features, and platforms.
📋 Recommendations
The analysis across product, market, users, and content would help you to rule out some hypotheses and confirm the cause. Depending on how well you are familiar with the data and the product, you might prioritize the user behavior section over the market one or product. That being said, regardless of your steps, you should link the affected metric (average tweets per user in our case) to the final recommendation.
When the analysis is complete, the next step is to provide your recommendation to teams, partners, and stakeholders based on your analysis. It can be something like:
Offer users a coupon to come back and try again (market cause)
Change or personalize the user experience for specific demographics or countries, for example (market cause)
Fix a bug in product feature or rollout (product cause)
Reconsider metric definition and calculation, for example, from tweets to tweets and visits (user behavior)
If you are being asked this question in an interview, congratulations! Walking through the steps and methodology described above will lead you to the right answer (or at least the right direction of product interpretation - and maybe a new job, in this theoretical case!).
If you are dealing with a similar question at work, think about the product+market+user model, and the explanation will be somewhere within that. Consider also the context and specifics of your product. A 20% decrease in usage can be quite big for a company with a larger user base, but not significant for a small sample.
Related publications:
Thanks for reading, everyone. Until next Wednesday!