Significance Level vs. Statistical Power in A/B testing - Issue 266

Statistics 101: Why analysts confuse significance with confidence, power with precision - and how to get it right.

Welcome to the Data Analysis Journal, a weekly newsletter about data science and analytics.

---

A very nice reader pointed out a typo in my A/B test Checklist, and it reminded me of a moment during a meetup (many years ago) when I was arguing with a data scientist about whether we should adopt Type I and Type II errors or Alpha and Beta in our statistical documentation.

My point was that if you studied statistics in China, Europe, or the USSR, most of the literature refers to Alpha and Beta, and referring to Type I or Type II errors is less common (at least it used to be). But in the U.S., I’ve noticed that most documentation, especially around A/B testing (and especially newer sources), mostly uses Type I and Type II errors. That used to confuse me - I could never remember which one was the false positive and which was the false negative.

So instead, I trained myself to just stick with Statistical Power and Confidence Level.

But even then, I kept noticing that during interviews and at work, people often refer to Power as Sensitivity, to p-value as if it were Power (which it’s not), or simply to Significance, which, depending on context, can also mean Confidence Interval (which some sources call Significance Level 🫠😂).

The reality is: there are only 4 key statistical terms we deal with in A/B testing, and yet we constantly confuse them or flip their definitions depending on context.

So, I decided to publish a quick refresher focused only on the core concepts: Significance Level, Confidence Level, Statistical Power, and Confidence Interval, and how to tell them apart, understand what they actually mean, and share their expected values.