Refresher on A/B Testing - Issue 258

Stats, tools, and trusted resources for doing experimentation right

Welcome to the Data Analysis Journal, a weekly newsletter about data science and analytics.

Yesterday, I ran a workshop on experimentation for a wonderful team of analysts who are transforming the creator economy. It hit me - I’ve never actually shared a proper introduction to A/B testing. Apparently, I just assume people are born knowing what statistical power or a p-value is.

So, today I decided to consolidate free tutorials and my bookmarked materials to help you get started with A/B testing and experimentation, including tools, key concepts to understand, free classes, and more.

---

Before we jump in, spreading the word about a couple of upcoming events next week:

🗓️ Data Science Salon returns to San Francisco:

Join DSS at the Amazon office next Wednesday for a meetup featuring speakers from Adobe and NextData. Teams will be talking all things data prep and management for AI and automation. Come hang out!

🗓️ I’ll be in Vegas next week, joining 2,000 marketers at MAU Vegas 2025 - mobile app AdTech and MarTech conference. If you work in mobile app analytics, come say hi! Use this link to get a discount: MAU Vegas 2025.

I have a dedicated experimentation section in my newsletter, but I realized most of my publications are geared toward experienced analysts, diving into tricky cases like accelerating experimentation, testing on low traffic, or running tests in marketplaces. What I don’t have is a simple introduction to A/B testing.

That’s partly because you can’t really start with experimentation without first understanding statistics. And I don’t think it’s helpful to talk about types of A/B tests with someone who doesn’t yet understand p-values or probability. (Explaining p-values isn’t easy either.)

So, I don’t believe there’s such a thing as a gentle introduction to A/B testing. And it’s definitely not something you just "pick up" on the job. Even basic familiarity with testing requires a solid foundation in probabilities. (Maybe that’s why it’s so hard to find analysts experienced with experimentation?).

There are lots of paid courses, online academies, and YouTube videos out there on experimentation. And Maven products. Below, I’m sharing my vetted, trusted, and personally bookmarked free sources on A/B testing. If you go through them step by step, in a phased approach, I guarantee you’ll be well-equipped to work with A/B tests.