10 Must-Know Concepts Every Analyst Should Know - Issue 290
From academia to industry - the core analytical principles we rely on every day.
This publication is for anyone transitioning into data analytics from another field, or, like me, who studied outside the U.S. and later had to unlearn why a “confidence interval” is actually uncertainty and not, you know… confidence.
I put together 10 foundational principles that shape how data scientists reason and communicate data. Think of them as mental shortcuts that help interpret the world through data. Some you might remember from university, others are probably unwritten rules passed down in Slack threads and analyses readouts — things every data scientist somehow just knows.
Warning - quick venting ahead. Looking back, the biggest challenge for me wasn’t methodologies or project complexity, but language. It’s frustrating how much slang, jargon, acronyms, and metaphors we use in everyday communication.
Here’s an actual slide from a monthly KPIs report:
“Paid channel miss of $230K driven by continued market headwinds”.
What does that even mean?? What are “market headwins”? Is that a good market or a bad one? Who talks like this? Why not simply say, “We made $230K less than planned from paid ads because the market stayed tough.”
Or another one: “Annual cohort mix change offset small gains in retention.”
What??? You understand every word, yet the whole thing makes zero sense. Again, why not just say, “We had more lower-value customers this year, so retention didn’t improve.”
But no. We have to make it intentionally hard to grasp, so no one actually knows what’s going on. Come on, I take this personally. Done venting.
Anyway, from language barriers to concepts that actually matter, below is my list of principles I expect every data scientist and analyst to know well. Most relate to critical thinking and statistics - concepts we use when analyzing A/B tests or running an analysis.
1. Occam’s Razor
In analytics, there are always many ways to solve a problem:
You can prove the relationship between feature usage and retention using simple regression, multiple linear regression, ridge regression, or elastic net regression. Which method should you use?
You can run a forecast using a simple moving average, ARIMA, Prophet, or temporal fusion transformer. How do you decide which method to use?
How to find a balance between simplicity and accuracy?
Occam’s Razor means that if a simpler story explains the numbers, stop there. The simplest explanation that fits the data is usually the best. Every parameter, filter, and feature you add introduces an assumption that may later break.
This principle is critical in analytics because the field naturally encourages complicating metrics, formulas, solutions, and dashboards. We need this principle to act as a governor. If the simpler method works or explains the data, stop there. Complexity must “pay rent” through better accuracy or clearer insight.
2. Simpson’s Paradox
Simpson’s paradox is a confusing trend where what appears to be a correlation in one direction may actually be a trend in another direction:
What it means: you have to account for movements within confounding variables (segments and subgroups). What seems to be a positive trend overall may, in fact, be a negative trend within each subgroup.
It’s one of the most dangerous traps in data science because it hides bias inside apparently clean averages.
For example, you might see an overall drop in conversion rate after a redesign, escalate it, and revert the change only to learn later that high-traffic countries improved while low-traffic ones dragged the total down.
Simpson’s Paradox is a warning: never trust averages or rolled-up metrics without context. If you measure A/B tests against ARPU, LTV, or Churn Rate - I guarantee you’re dealing with Simpson’s Paradox often than you think. Segment every sample into groups, then each group into subgroups. Check distributions by geography, platform, user type, and time.
3. Base-Rate Fallacy
I suspect the base-rate fallacy is why so many growth marketers are obsessed with measuring top-of-funnel tests against ARPU or LTV, or with creating overly complex rates. It sounds cool. Scientific. Like it should be. But simply measuring against converted customers is likely wrong. Too simple.
The base-rate fallacy happens when we ignore baselines and focus only on specific or recent details. For example, if 60% of users churned last month, a typical “churn predictor” without historical data might show a 99% risk for half your user base. In reality, such a high churn rate could be expected for that month. It could be because most users reached their end period, it was a low season, or a promotional campaign had ended.
Or here’s another very good example:
Analysts must start from priors: how frequent is the event you’re predicting? That’s why we need baselines and historical data.
4. Law of Large Numbers
You run a localized test with 5K users and received, on average, 100 event logs per user per day from a new page. You remove the Control and launch a new page for all 50M users. What are the new expected event logs per user per day?
