Methods To Accelerate A/B Testing - Issue 202

Strategies and statistical methods to increase test velocity.

Last year, I published Embracing the New Era of Accelerated Testing, which was both somewhat controversial and emotional to write. 

As a statistician trained to adapt academic principles to the fast-paced tech environment where nothing is trusted, I had to acknowledge that the concepts we were taught at school have become outdated and no longer serve us well.  

The new generation of tools has accelerated the speed of product delivery. Every aspect of mobile/web development, including design, QA, and research, now runs twice as fast as it did a few years ago. Tools like Split, Superwall, Adapty, and even native Apple solutions offer incredible capabilities. Today, we have the technology to iterate "on the fly" by continuously shipping and optimizing features. 

However, A/B testing practices and frameworks have remained the same, creating a gap between how fast the team is ready to move, how much trust we put in the data we receive, and how quickly we decipher its signals.

Teams want to run more tests - faster and more efficiently.

Let’s discuss today what you can do to increase test velocity. What statistical methods and solutions are available for you to leverage to speed up testing, and how can you strike a balance between trust and speed?

Analytics is accelerating. Gear up.

Duolingo is running "a few hundred experiments simultaneously.” At Pinterest and Uber, over 1,000 experiments are active at any given time.