Why it’s wrong to run A/B test for too long - Issue 213

The risks of extending the A/B test timeline or why duration matters.

I’ve talked a lot about why you shouldn’t stop A/B tests early and what the caveats are to disregarding the test significance. 

However, one topic I haven't yet addressed in my newsletter is why it's also wrong to run an A/B test for too long. 

In other words:

• if the test proved the hypothesis and

• the winner is truly making a difference and 

• the observed lift didn’t occur by chance, 

Why does it matter if the test runs for 2 weeks or 6 months? If the winner is indeed true, shouldn't the longer timeline confirm it?

Not quite. Let's dive into this topic today.