Rethinking A/B Testing for B2B and SaaS - Issue 274

Lessons from StatSig on best practices of designing and running experiments in B2B

Welcome to the Data Analysis Journal, a weekly newsletter about data science and analytics.

---

When we talk about experimentation, we usually mean it in the context of B2C products. It’s rare to find examples or case studies on A/B testing for B2B or SaaS, where onboarding is sales-driven, user samples are small, there’s little variance, and every customer has a dedicated account manager providing white-glove support. What’s there to test?

I used to think that if there was no self-service option for customers, there wasn’t much to experiment with.

I was so wrong! There’s a lot of experimentation happening in B2B, it’s just different.

I haven’t run A/B tests for B2B or SaaS yet, so when I saw Statsig’s talk at this year’s Data Council on experimentation in B2B, I was intrigued. It ended up being my second-favorite presentation (after Roblox’s session on causal inference).

Today, I want to share my takeaways and learnings from that talk. While the principles of A/B testing never change, running experiments in the B2B world comes with unique challenges and some surprising advantages. Let’s break them down.