Discover more from Data Analysis Journal
Growth, Loops, And Some Hard Truths - A Recap Of Amplitude Cohort 2022 - Issue 120
Great speakers, tasty food, good topics, and a puzzling lack of analysis
Welcome to a free edition of Data Analysis Journal, an advice column about data and product analytics. If you’re not a paid subscriber yet, here’s what you missed this month:
Freemium vs Free Trial Analytics - what is the difference between trials vs freemium? The top metrics behind each model, and the challenges in measuring and reporting the impact and growth of free trials vs freemium vs reverse trials.
How To Pass A Technical Assignment For Senior Data Scientist Position - an example of a technical assignment completed in Python for a Senior Data Scientist position at one of the biggest Bay Area companies.
Joining Two Tables That Are Not Related By A Foreign Key Relationship - how to connect two tables that are not related by a foreign key relationship in SQL (for cases when you can’t use UNION)
Last week I had the privilege to join an Amplitude event called Cohort 2022 where the team announced new features, presented a new product report and covered patterns and trends in today’s PLG universe.
The theme of the event was focused around unlocking growth, and growth experts shared their takes on the current situation, their learnings, and 2023 growth strategies.
While I enjoyed the speaker talks, panels, and shared insights, as an analyst, I was disappointed with the overall agenda, the lack of data expertise representation and the trend Amplitude is after. I don’t mean to be negative or speak poorly of Amplitude, but I can’t help entering a function like this without an analyst’s mindset. It’s what I do, and today’s journal will speak to the event through the lens of a data analyst.
Leading in a down market and introducing a new product report
My favorite part of the event was Spenser Skates’s (CEO, Amplitude) opening speech Leading in a Down Market. He addressed the recession and empowered all of us by saying that the best companies were launched during the recession. Isn’t it the best way to kick off the growth event?
💡 Make sure to check out their Product Report 2022. Insights are fascinating. Some of them echo their previous trends analysis I covered a few months ago - Party Like It's 2019: Product Trends Analysis In A Post-2020 World.
Then Amplitude introduced two new features:
⭐ Profile Connect: Amplitude + Snowflake lets you run analysis in Amplitude using data you store in Snowflake. They offered integration with Snowflake before, but this meant to be a new “seamless connection”.
⭐ The campaign reporting tool aims to explain how acquisition investments drive your product growth. It lets you analyze your acquisition channels, and leverage multi-touch attribution, impressions, and ad integrations. It’s good because now your headache about half of the source data being NULL or missing or not making sense will be everyone else’s problem, too.
And everything else (well, almost everything)
The event description teased us with insights from the best in product, growth, and data. Every talk was focused on growth being cross-functional and a result of merging product, marketing, and data analytics.
📢 Marketing was nicely represented by a strong panel moderated by Ashley Stepien (VP of Global Marketing at Amplitude) and a touching discussion between two CMOs - Carilu Dietrich and Tifenn Kwan (I am a true Tifenn fan btw. Love her energy).
🌱 Product topics and initiates were covered within the growth theme in Justin Bauer’s (CPO, Amplitude) talk and during the panel of experts from Canva, Notion, Webflow, and Postman, who talked about their Growth team structure, mission, priorities, and learnings. Then Adam Fishman killed with his amazing talk on strategies and specifics of building strong growth teams.
📊 Data analytics lurked here and there but was not the main focus of the event. I would love to see more leaders in the data and analytics space contributing to the growth discussions and questions raised. May I recommend a good Head of Analytics to join your next Amplify 2023 event ☀️?
Gentle kidding aside, I worry that the lack of data experts' representation at such events could lead some to believe that unlocking growth doesn’t require quantitative analysis anymore. Or that data inference and precision overall become unnecessary and overrated in PLG.
Duh, let’s!
At this event, I was hoping to hear examples of how companies unlock growth using Amplitude analytics. How it empowers (and magnifies) the track → insight → act framework by helping product analytic teams to:
Prove the correlation and causation between various user activity patterns.
Locate those patterns in the first place in order to define user cohorts that they actively recommend.
Find approaches to segment customers to offer them personalized messaging and offerings.
Run regression analysis to confirm the relationship between the usage of product features and user engagement.
Develop a feature matrix to understand the volume of usage and ROI of products.
Take advantage of their forecasts and be prepared for the expected seasonality decline.
Examine true or not-so-true data spikes (which are terribly easy to get tricked by in Amplitude).
I was hoping to hear case studies on how product analytics empower product and marketing teams to scale growth by reading MoM and YoY activation rate changes, caveats to be aware of in user engagement metrics monitoring and calculation, or signals to watch to optimize the onboarding funnel to decrease time to action. Based on my experience, these projects are most successful when product owners partner with data analysts.
Exploratory vs precise
If you’ve kept an eye on Amplitude over the last few years, you may have noticed that it has become very “Reforgy”. It’s a good thing, and we all should get excited, but I wish it wouldn’t come at the cost of cutting short data quality, data handling, and analysis.
I have been using Amplitude for more than 6 years at 3 companies, and as much as I like it, enjoy its intuitive usage, appreciate its features, and recommend it (!), I don’t trust its analytics for any crucial decision-making. Especially regarding growth-related insights.
Event-based nature analytics may be misleading (this is what is used for your Top of Funnel data). If you are using Amplitude for monitoring your activation, time-to-action, and retention, there is a high chance you may read the data wrong for two reasons:
1. Activity data is at the mercy of the unreliable nature of user activity events
Analyzing app open, session start, screen view, and login in Amplitude is good for understanding user flows, and the overall volume of events on a platform, but trusting the proportions, which are needed for activation rate deep dive, can be tricky.
Here is one example of how Amplitude events may mislead you. In a nutshell, the author goes through SDK Amplitude integration explaining how the silent notification system wakes up your app in the background and initiates sending analytic events and attributes, like Session Start. It means you will count activity for users who did not open your app at all.
There are many other cases when the data you see (either in Amplitude or Snowflake) might not be correct. Having an analyst by your side will ensure the report you create is correct and relevant.
2. Exploratory data analysis is not a deep-dive analytics
Amplitude features are more of an “exploratory” type that is not meant to be precise. But you still bet your (and your company’s) money on it.
Most of its functionality - funnels, segmentation, and retention charts - provide high-level analytics. But that’s not up to the Reforge standard. Reforge developed its growth series, strategy, and monetization frameworks on a thorough deep-dive analysis that requires high data precision and trust. The amplitude trend to be “Reforgy” does not meet their current functionality:
Its analytics do not offer a weighted average for you to get your metrics calculations precise.
It does not let you see YoY or MoM retention graph changes to understand how you grow.
Their trends and forecasts are nice but directional.
There is no correlation or scatter visualizations that would unlock regression analysis.
There are no heat maps that would help you to see what works or doesn’t work in your product.
And while you will be able to get flavors of these insights directly in Amplitude, you need an analyst to take it further.
For example, as mentioned during the event, Airbnb increased CVR by 2.5 after learning that good photos drive more conversions. Or DoorDash doubled down on the precise wait time estimation. How can this be derived and confirmed in Amplitude? By using their new Engagement matrix feature? Pathfinders? Grouping and segmenting funnels? As someone who spent a decade extracting and developing such insights, I always have to supplement Amplitude with Excel, SQL, and some type of visualization tool to understand factors that drive growth.
To be fair, they do have great features that I am obsessed with and use every day. I LOVE their Root cause analysis which saves me a lot of time. I like their User composition, and find it quite accurate. I am taking advantage of their Cohorts. I am dependent on their Hourly and Real-time segmentation data and use it for early smoke tests, catching data issues with small rollouts, etc. I am grateful for their Funnels which I have been using for A/B tests data monitoring for ages. Their significance and confidence features have helped me estimate how long the test will be running, and how much I can trust the result. I like their Reports and Spaces, and potentially see it as a house for analytical documentation.
Overall, I truly believe Amplitude is the strongest product analytics tool on the market right now. But I wish they would invest more in data correctness and precision as well because the data quality is the foundation of analytics. And if this foundation is missing, it can endanger the entire structure.
Subscribe to Data Analysis Journal
Where product, data science, and analytics intersect. Trusted by tens of thousands of data scientists around the world
This is a really good analysis Olga, thanks for sharing. Amplitude is indeed investing in data quality initiatives (https://info.amplitude.com/amplitude-iteratively) and I believe, they will gradually get there in terms of building trust in the data they collect.
And yes, I totally agree that Amplitude (as well as companies offering similar tools) should do more to appeal to the analyst persona -- they're all lagging behind at the moment.
Well done, Olga! Appreciate how objective and direct you are at the same time. Hope, they see it.