How Much Is an A/B Test Worth? - Issue 323
A new practical framework for deciding when to test, how much data to collect, and when the evidence is worth the cost.
Most companies understand that experimentation is important. The more tests you run, the faster your organization learns, and the better your data-driven decisions should become.
They also know (and I take some pride in that) that A/B tests must be conducted properly and follow clear statistical protocols. I have spent years writing about the importance of statistical significance, statistical power, and p values. I have also warned teams that even when they do everything right, still, 70%-92% of A/B tests will be inconclusive or misleading - showing an apparent lift when the true effect is neutral or even negative.
This is why many scientists and analysts remain skeptical of A/B testing:
I used to be in that group too.
Many teams do not invest in proper experimentation, and they test everything. Inaccurately. Then, they become convinced that shorter onboarding flows perform better than longer ones, hard paywalls increase conversions, or dark mode improves engagement, all based on that one test they ran 2 years ago. They keep sending me a slide / case study with 2 numbers for Control and Variant, showing some % lift. I’m not able to replicate it, validate it, pull those users to check distribution, or do anything with it, and yet I’m supposed to treat it as a proven causal learning? (Onboarding tests are particularly good examples - they’re the hardest experiments to conduct, and if you’re measuring them using typical events setup via Amplitude or Mixpanel, I’m confident your test read is wrong.)
I am less skeptical of A/B testing now because I have seen what experimentation looks like when it is done at scale. Best-in-class companies often run the same test many times across different traffic levels, seasons, audiences, budgets, and product conditions. When teams can separate the effect of the change itself from the effects of seasonality, audience mix, spend, or other conditions, they can be much more confident that the result is real and repeatable.
This is one reason companies are now doubling down on accelerated experimentation, scaling from 10 tests per month to 100 or more:
If most tests don’t give a trusted answer, teams need an environment where they can quickly identify the few tests that produce clear results and act on them.
But scaling experimentation creates another problem - cost.
Once teams begin running experiments at scale, things become chaotic. They often cannot tell whether they are running too many tests, too few tests, or tests that are much larger and more complicated than necessary. They simply plug in common statistical rules like 80% power and p < 0.05, without connecting those decisions to revenue, cost, or business risk.
What it means is that some tests can run for months. They take traffic away from more urgent or valuable experiments and slow down the entire testing program. A team may increase from 10 tests per month to 100, yet complete fewer than 2.
Methodologically, the team may be doing everything right:
You cannot stop a test before it reaches the required sample size or statistical threshold.
At the same time, you cannot realistically adjust the confidence level for every experiment - accepting 90% confidence for one test, 85% for another, and 95% for another, when hundreds of tests are running.
So what should teams do?
Data scientists and researchers from Amazon, MIT, Stanford, and Columbia collaborated to solve this exact problem. A few months ago, they published a paper asking: when companies scale experimentation, how should they decide how much each experiment is actually worth? In other words, how do you optimize A/B tests for business value rather than sample size and statistical conventions?
Below, I will walk through their research and explain how their framework helps teams make experimentation decisions based on economics rather than statistical rules alone.
