How To Run An A/B Testing On Low Traffic - Issue 181
Strategy and solutions for effective A/B Tests with small samples
Welcome to the Data Analysis Journal, a weekly newsletter about data science and analytics.
Before we dive in, I wanted to ask a favor to fill out this brief survey (And thank you so much to those who have already done it! You are the best! ✨). As my audience grows, I’d like to better understand how my journal can stay relevant and bring more value.
The most common question I have received from my readers has been how to do A/B testing with very low traffic.
There are no specific rules or requirements dictating when to start A/B testing or the minimum number of users necessary to launch a test. There are various opinions on how to approach experimentation with small sample sizes, ranging from more conservative “You should not run an A/B test until you reach 30,000 users per variant period” to the more progressive “It’s fine, just adjust your thresholds.”
Today, I will share my guide on the procedure, specifics, and caveats of experimenting with low traffic. I will cover topics such as the minimum number of users needed to launch an A/B test and offer suggestions on increasing confidence and trust in small sample tests.
I’ll present it in a Q&A format to ensure I address all the questions I received from readers regarding low-traffic tests (I apologize for the delay in addressing this):
What is the minimum number of users needed in a sample size for an A/B test?
Can we run a test on a very small sample (~50 users in Variant per day) and wait longer to reach significance?
What other factors should we consider to ensure we make the right decision?
How do you figure out the trade-off between confidence and test timeline? For example, can we test small samples with an 85% confidence instead of 95%? Will the test run faster?
What are some rules or testing limitations in low traffic to address to increase statistical rigor?
If you need a refresher on the A/B testing concept or procedure, read the A/B Test Checklist.
“When running online experiments, getting numbers is easy; getting numbers you can trust is hard.” - Online Experiments: Practical Lessons, Microsoft.
Q1: Is there a set rule for the minimum number of users required to launch an A/B test?
Keep reading with a 7-day free trial
Subscribe to Data Analysis Journal to keep reading this post and get 7 days of free access to the full post archives.