Sunday Q&A: How Teams Fail Themselves - Issue 275
A reader’s story about impossible expectations, the path to becoming an analyst, and actionable retention reporting.
Welcome to the Sunday Q&A: your questions, real stories, and the occasional rant.
Previous Q&As:
Sunday Q&A: Personas, Growth Projections, and Why SQL Still Matters
Sunday Q&A: Maven Classes, Analytical Thinking, and Best BI Tool
In this Q&A, I dive into the technicalities of retention reporting, the journey of becoming an analyst, and share a story about how teams underestimate analytics and the art of setting people up to fail.
Retention for e-commerce
“Hi Olga, would love your input on the retention. I’m taking ownership of our reporting and want to reset the key metrics teams have been tracking. Our previous retention setup was neither actionable nor accurate, so I’m trying to establish a new logic, using your publications, but I’m stuck on how to frame retention for our product.
We’re an e-commerce company with a long history of users. Roughly every 5th person has or had an account with us, but only a small % ever transacted, and even fewer came back. If I anchor on signup, retention looks terrible - like a flat 95% drop-off that doesn’t move, even though I know we’ve made improvements that should show up somewhere.
How would you approach making retention reporting more meaningful and actionable in this case?”
Related: How to measure cohort retention and Refresher on Retention
First of all, don’t anchor on signups. I write a lot about subscription-based products, and yes, they should start with signups, since their funnels are narrower and more sensitive. But for e-commerce (B2C, I assume), there’s usually a big gap between signup and the first transaction. My rule of thumb (feel free to challenge it) is this: if fewer than 5% of signups do X, and X represents your core product value, then you shouldn’t start with signups for retention. It won’t be helpful or actionable.
In your case, I’d start with one of these (or both):
Users who activated (Searched? Completed an integration? Opened a wallet? Created a “pin”?)
Users who successfully completed their first transaction.
From there, segment retention by customer group:
New users
Activated or 1st-transaction users
Returning (not new) users with 10+ transactions
On top of that, I’d also segment by (a) the type of transactions and (b) the number of transactions to better understand user personas. When you see different retention thresholds, you know you're doing it right.
Remember, if you want to measure retention over time, be careful with the unbounded method. Older groups will naturally look stronger, because they’ve had more time to come back. For historical retention reporting, I always recommend using N-Day, but with a large enough time window (a month or longer). If you do cohorted, any way works.
Figuring out the data path
“Still trying to figure out, in part by reading your articles, what I really am. Am I really an analyst or a PM, or DBA, or DE, or...? So far it seems to be a mix.”
You don’t really know who you are until you’ve tried on all the hats. And the best place to do that - a very early-stage startup or a small company where it’s expected to be the DE + DBA + Analyst + “whatever the CEO needs help with before lunch”. Odds are, that means creating a deck with 42 versions of the same bar chart, just so one of them makes a flat curve look slightly less… flat.
All these roles touch data, but they produce different outcomes and require different skill sets. So, subjectively:
If you’re quick to pick up SQL and Python, feel more comfortable with structured projects and clear guidelines, and can troubleshoot small but annoying issues, you’re probably a data engineer or DBA.
If you get a thrill from puzzles and brain teasers and have a decent memory for numbers, you’re most likely an analyst.
If you struggle with math and stats, you might be better suited as a PM - because you're likely to fail at other roles.
Analytics becomes very specialized and nuanced - Which Analyst Are You?
Stop setting analysts up to fail
Today, I want to share a story I received from a reader a few months ago. It highlights a lot: the reality of working at a startup, where role expectations are undefined and always shifting. What happens when teams fail to invest in onboarding and documentation, and how cutting corners leads to a complete lack of organizational readiness to leverage data.
“Got an offer from a startup. They had a small data team, but I was assigned to a business team which never had any analytics support, so all the dashboards etc had to be built from scratch, also the stakeholders had no idea how to interact with a data person, and my manager (interim head of data) was very overstretched.
I was fired from the startup last week, after 2.8 months of employment.
Main reason given was that I was good at the role they initially hired me for, but the scope of the role changed as they needed a more senior person, while I was good with the sql, building dashboards whenever something was asked of me.
But in reality expected me to absorb all the business knowledge within 1-2 months, identify on my own, the issues that the Ops team I was embedded in had and proactively provide data insights, to help the ops team with ops processes so that I could fill in my knowledge gaps, quickly learn GIT and DBT to start adding to the data layer, provide the Ops team with visualisations they did not know they needed, form strong relationships with the tech team, scope features that could help us add more data points where we had gaps and work with engineering to get those features in place, try to really distill what the stakeholders were asking for (a major piece was weekly reporting on that business area for upper management that my main stakeholders wanted to improve but had no clear idea how to and my manager wanted me to basically do it but at the same time she did not agree with what the stakeholders were asking for and wanted me to come up with something and then convince the stakeholder of that and essentially take the load off her, propose projects to improve various parts of the business area I was in like pricing etc. Basically in their words during the call "I was a sports car and they were looking for a SUV".
I could've done all this after 4-5 months there but I did not have the experience and the skills to be able to demonstrate all of this within 2 months of joining the business, especially stakeholder management which I know I am weak at especially when I am still coming to grips with the data and the business and scoping how to collect the data we needed when I did not even know I needed to first understand how the different tech systems spoke to each other in terms of data before being able to scope the features needed, all this was just assumed I will figure out on my own in their unstructured, cross-functional environment. I was already overstretched with learning the database, churning out dashboard after dashboard, dealing with adhoc requests and multiple meetings every week such that I was working extra hours and even some weekends, and doing the above on top of all that within 2 months was simply not possible for me. "
I already responded to the author who was fired for not doing the impossible. I’ll quote myself from Mastering Critical Thinking: How to Improve Your Analytical Skills, published over a year ago:
“...For my team, I don’t allow analysts to own any reporting until they’ve been with the company for at least 6 months. It takes that long, on average, to grasp the data flows, user interactions, data quality, and dependencies.”
And that’s for mature companies - with proper onboarding, documentation, and “running“ analytics in place.
To the author: don’t let poor onboarding, bad data management, and lack of support shake your confidence.
To the rest of us: let’s do better.
—---------------------------
Thanks for reading, everyone!