10 Experiments Every Data Team Should Run - Issue 288

Top must-try experiments every data scientist should know - lessons from StatSig Summit.

Welcome to the Data Analysis Journal, a weekly newsletter about data science and analytics.

---

Today I want to share a recap and summary of a recent talk by Liz Obermaie, a former Meta data scientist now at StatSig, at the summit in San Francisco

I’ll admit, I was skeptical first. The title felt a bit over the top, but after watching the recap, I was impressed. The content is practical and well-structured. Nothing I haven’t covered in my newsletter before, but Liz explained it so clearly that I had to save it - and share it with you.

Below is my summary and commentary of her talk — the 10 experiments every data team should run. Some things I don’t agree with, but overall, it’s a very decent step into data science 2.0.

All Models Are Wrong (But Some Are Useful)

Quoting George Box, “All models are wrong, but some are useful”, is actually a great way to introduce experimentation. It’s controversial because people focus on the “wrong” part. But what it really means is that every model involves uncertainty and assumptions. There’s always a tradeoff between the assumptions you make and how confident you can be in the results.

That tradeoff sits at the core of experimentation. As data scientists, we tend to overcomplicate this. We want to explain every assumption in detail, while business stakeholders just want to know: What should we do?

Why randomized controlled trials matter

There are many types of experiments (behavioral studies, quasi-experiments, difference-in-differences, lots of user research qualitative techniques), but this presentation was mostly about randomized controlled trials (RCTs), where randomization does the heavy lifting - exactly what happens during an A/B test.

Randomization is very important - it balances test confounders, both known and unknown, so you don’t have to model every possible bias. That frees up your mental bandwidth to focus on the real work: edge cases, communication, and stakeholder alignment, rather than defending your assumptions. In other words, if you’re unable to randomize your traffic, you still can run experiments, but analyzing and interpreting them right will take 10x more effort and time - you would need to model the lift separately for every possible attribute.

Keep in mind that randomization can happen at different units - sometimes it’s not users, but sessions, queries, or even servers. The right choice depends on the assumptions you can safely make.

Part 1: Basic Experiments

These are simple but foundational. Most organizations run them early and often.

1. Standard A/B Growth Test

This is the classic marketing experiment. You test variations of a message, call-to-action, or landing page.