How To Measure Product Adoption - Issue 27
Product adoption, data democratization, and DS trends in 2021
 |
Hello analysts and welcome to a once-a-month-free-edition of the Data Analysis Journal newsletter, where I write about data analysis, data science, and business intelligence.
If you’re not a paid subscriber, last week you missed a User Activity Analysis in SQL (one of the most common SQL problems often asked on technical interviews) and my walkthrough on analyzing user activity by applying SQL RANK() function.
✨ In today’s newsletter:
How to deploy ML models to production and why 87% of them never make it.
How to measure the success of a new feature or a product.
What data democratization is, why it is important, and how to develop data culture at your company.
SQLserver 101 or how to get started with SSMS basics.
2021 digital marketing trends report.
*For paying subscribers, apologies if you received this newsletter twice. Big mistake happened while figuring Substack magic.
🏆 Nailed It
Be prepared for your next interview
How to measure the success of a new feature release or run a product adoption
One of the most common challenges in product analytics is measuring the success of a new feature release or analyze product adoption - how users discover it, how they use it, and how often. Simple questions can become quite complex when you start breaking these down on a timeline, user segments, or engagement layers.
For subscription-based products or services, measuring product adoption is essential. A high adoption rate leads to increased:
revenue
LTV
trial to subscribed conversion
free to paid plan conversion
It’s a lot of pressure on ensuring you tackle the measurements correctly. Here are the top 2 metrics to start with:
Adoption rate:
% users who used a new feature / all existing usersThis metric is useless without applying a specific timeline. To get the Adoption rate for December, users who used the feature for the first time any day in December divide by the total number of users you have, on the last day of the month.
Time-to-Action:
avg time for a new customer to use an existing featureor
avg time for an existing customer to use a new feature for the first time.❗Things to remember:
The more users are engaged with a product, the faster they will discover new features, and the more often that they will use them. This is a big caveat if you are running an A/B test for a not randomly distributed audience. The measurement of feature discovery and usage will be skewed and potentially overreported.
If your feature of the product is complex or requires customer learning, you might have to introduce the Time-to-Adopt metric in addition to the Time-to-Action to measure how much time it takes for users (either new or existing) to start using a new feature.
Another secondary metric to look at is Duration - how long users keep using the feature after discovering it.
If you are launching a new product and working on setting top-level metrics, here is a good guide from ProductLed in order to best understand your Northstar metrics and KPIs measurement.
🎓 Level Up
Refine your skills, develop, and grow
Today, I’d like to share today a new publication from Data-led Academy about developing data culture at your company and how to promote data democratization - “enable teams to work with data, feel comfortable talking about data, and as a result, make data-led decisions.”
Here are some interesting takeaways:
The article covers different ways teams are using data for:
Marketing works with data to create engaging, better-converting content
Growth works with data to run experiments and deliver personalized experiences
Product and Engineering work with data to build features that are actually used and kill the ones that are not
Support works with data to deliver faster resolution (by seeing what a user has done or not done inside a product)
Customer Success works with data to deliver a better customer experience (by asking them the right questions based on usage patterns)
Sales works with data to identify prospects that are likely to convert (by looking at the actions they have performed during the free trial)
The most common data challenges include not having the access to the data, lack of trust in data, no appropriate skills to find the answer to questions, not robust or helpful analytic tools, or simply that other data experts are too busy to help.
Read What is Data Democratization? to learn how to tackle these challenges and be an important part of creating a cultural shift at your company towards data democratization.
🔥 What’s new this week
2021 Data Science Trends report 📈 covers DS trends we should expect to see in 2021:
Operations - DataOps and MLOps which are responsible for bringing AI models and theoretical results into the production.
Data democratization - technologies and services that make data easily accessible to everyone in the company for decision making.
Hyperautomation - combining AI, analytics, and IoT that augments Robotic Process Automation (RPA) to automate business tasks and improve efficiency.
Low-code no-code platforms - automated ML services developed for non-technical experts.
Cloud-native ML - flexible architecture that “scales according to its usage.”
Explainable and responsible AI - regulations on how AI is trained and developed, making it more ethical and trusted.
2021 Digital Marketing Trends report 📈 does a good recap of 2020 with setting a projection of some trends for digital marketing:
Video streaming increased by 12% in the first week of quarantine in the U.S.
In March 2020, when the lockdowns and social distancing protocols first began, Facebook’s analytics department reported a 50% surge in messaging. Similarly, WhatsApp saw a 40% lift in usage” And you know the story with TikTok.
In 2021, people will spend an average of 100 minutes per day watching online videos.
Marketers will likely continue to pursue deeper personalization and more localized targeting — trading reach for relevance.
PostgreSQL is the DBMS of the Year 2020 🎉
The second place is the SQLserver (Azure SQL DB built on top of SQLserver). The third place is MongoDB.
Basics of SQL Server Management Studio (SSMS) – Part 1 ⚒️:
Basics of SQL Server, how to install it and get started. Must-know terminology, prerequisites, and basic actions.
📚 Weekend Longread
How to put machine learning models into production
The goal of building a machine learning model is to solve a problem, and a machine learning model can only do so when it is in production and actively in use by consumers. As such, model deployment is as important as model building.
BUT. 87% of data science projects never make it to production.
There can be a “disconnect between IT and data science. IT tends to stay focused on making things available and stable. They want uptime at all costs. Data scientists, on the other hand, are focused on iteration and experimentation. They want to break things.”
Bridging the gap between those two worlds is key to ensuring you have a good model and can actually put it into production.
🍸 Drink and Mingle
Upcoming free events, meetups, talk, and webinars.
Jan 13, Galvanize: Intro to Python: Part 2
Jan 14, WWC London: ML Workshop: Hunting for Pulsars
Jan 14, Redis: Introduction to Redis for Data
Jan 18, Galvanize: Intro to SQL and RDBMS
Jan 21, Recurly: How Finance and Operations Leaders Can Use Data to Unlock Growth
Jan 21, Galvanize: How to Approach a Python Coding Challenge
🙏 Do Some Good.
It pays back.
I am one of the members and volunteers of Women Who Code (WWCode) - the largest and most active community for technical women in the world. WWCode award-winning programs support over 230,000 members around the globe to level up in their technical careers, every step of the way. They have produced 10,000 training, workshops, hackathons, conferences, and more. If you are looking for an organization to donate to, please support WWCode work. Donate here.
Thanks for reading, everyone ❤️. Until next Wednesday!
Hey Olga. This was a great read. I loved the Product adoption metrics part which helped me in one of my recent interviews. Thank you so much for this blog and all the great reads and events you have been sharing. It has definitely helped me in refining my analytical skills. :)
I came across this case study question as I am learning more about how A/B tests help in analyzing the success and engagement of new product feature releases.
Say you're conducting an A/B test for a new product feature in a B2C company, and your test cycle is typically 5 weeks. Two weeks has passed so far and your results are as follows:
Week 1: The conversion rate of your Control group is higher
Week 2: The conversion rate of your Test group is higher
Now the executive leadership team wants to know how the test is performing since they want to decide whether to release the feature or not. The leadership team is not willing to wait for the test to run for 5 whole weeks, since they would like to take this business decision a lot sooner;
The question is how will you present you analysis to the leadership team and how will you explain the change in conversion rate between your test and control groups?
From your experience, I would love to hear your thoughts on this especially on how you would present your analysis to the executive team and what other data points would you take into account while answering follow ups from your leadership team.
Looking forward to hearing from you. :)