Frequentist vs. Bayesian: Which Method Should You Choose for Your A/B Testing? - Issue 224

Learn the pros, cons, and use cases for Frequentist and Bayesian approaches to ensure reliable and trusted A/B test results.

I don’t usually clutter your inboxes on a Sunday, but I have one more quick topic to share with you this month.

Last week, a colleague asked me which type of A/B test he should run -  Bayesian or Frequentist. I searched for an article that explains the use cases but couldn’t find anything that doesn't dive into Hypothesis Testing Theory over 10 pages while still being helpful. So, I decided to write my own today.

Let’s talk about the types of A/B testing and which method to choose for different use cases. I know it’s Sunday, so I’ll keep it short and sweet.

A quick recap on hypothesis test statistics:

An A/B test is a form of hypothesis testing. There are many types of A/B tests, and at work, we typically apply A/A, A/B, Split, or Multivariate testing.

Frequentist and Bayesian approaches are different methods of interpreting A/B test results. These are NOT types of A/B tests you can choose but rather approaches to interpreting the test results.

While most articles refer to Frequentist and Bayesian methods as “ways to read test data,” I prefer to think of them as different schools or, better yet, movements in statistics to handle experimentation.

Frequentist statistics 

If you graduated from any statistics class, you were likely taught the Frequentist method for interpreting A/B test data. This is primarily what academia has taught us. The Frequentist approach makes inferences about the test results using only data from the current experiment, without any prior context about user behavior. All parameters (mean, median, variance, etc.) and distributions used for A/B test analysis are considered fixed.