Refresher on Experimentation - Issue 294

A consolidated guide to statistics, tools, best practices, and advanced experimentation

Welcome to my Data Analytics Journal, where I write about data science and analytics.

Thank you to everyone who came to Hex Magic Night in New York last night. It was great meeting so many of you and hearing about the magic you create with analytics.

I hope the talk was useful as you think ahead to 2026 - the trends, the risks, and what it will actually take to be ready.

Next Tuesday, I’ll be in San Francisco with OpenAI and Fivetran for the same conversation on preparing for 2026. You can register here: Hex Magic Night - San Francisco. See you there!

As has become a tradition, I end the year with a refresher: one single, consolidated guide that brings together the most important definitions, reporting best practices, example dashboards, and key principles - all in one place.

Last year, I published these three:

1. Refresher on Retention

2. Refresher on Statistics

3. Refresher on SQL for Data Analysis

This week, I’m sharing a refresher on A/B Testing. I believe I published one before, and I also have a dedicated experimentation section in my newsletter. This time, I expanded it with more resources to help you properly get started with A/B testing. It covers tools, core concepts, free classes, and more.

I don’t believe in a “gentle introduction” to A/B testing. This is not something you simply pick up on the job. Even a basic understanding requires a solid foundation in probability and p-values. And p-values are not easy to explain. You can’t make them “intuitive” or “easy to grasp” if someone doesn’t understand distributions.

Yes, modern testing tools, calculators, and dashboards do a lot of the mechanical work for us. That helps, until you run into real-world edge cases: low traffic, uneven splits, phased rollouts, very small effects, bad randomization, or pressure to speed up or the need to scale experiments. In those cases, a solid understanding of statistics really matters.

Below, I’m sharing my vetted, trusted, and personally bookmarked sources on A/B testing. If you go through them step by step, in a phased way, you’ll be well equipped to work with A/B tests, even in more difficult cases.

Bayesianism vs Frequentism

New to A/B testing? Start here:

Online course and course recaps

• Udacity free online course - Introduction to A/B Testing.

• Vwo intro course A/B testing with more focus on marketing - What is A/B Testing? Used to be free. Requires a trial now.

• A Summary of Udacity A/B Testing Course - a good one.

• Synthesis from Udacity A/B Testing course

• Udacity A/B Testing Course Notes

Refreshers and intro guides

• A Refresher on A/B Testing from Harvard Business Review

• An Intuitive (and Short) Explanation of Bayes’ Theorem

• Twymans Law from Ron Kohavi

• Significance Level vs. Statistical Power in A/B testing

• An introduction to A/B testing and optimization from Yaniv Navot

• Frequentist vs. Bayesian: Which Method Should You Choose for Your A/B Testing?

From theory to practice

When experiments don’t work: