A short guide to product experimentation steps and process.
Hey, if the significance level is the probability of detecting an effect when none exists, shouldn't the benchmark be 5% instead? The guide states 95%, which I assume should actually refer to the confidence level. Thanks for clarification :)
Ha! That is a confidence level indeed, it does sound confusing. Updated.
Also, this is an outdated guide. I updated it last year - https://dataanalysis.substack.com/p/ab-test-checklist-issue-233
Good callout, thank you, I need to write a refresher to clarify all 4 concepts:
1. Significance level - 5%
2. Confidence level - 95%
3. Statistical power - 80%
4. Confidence interval - depends, e.g., 2%-4%
Hey, if the significance level is the probability of detecting an effect when none exists, shouldn't the benchmark be 5% instead? The guide states 95%, which I assume should actually refer to the confidence level. Thanks for clarification :)
Ha! That is a confidence level indeed, it does sound confusing. Updated.
Also, this is an outdated guide. I updated it last year - https://dataanalysis.substack.com/p/ab-test-checklist-issue-233
Good callout, thank you, I need to write a refresher to clarify all 4 concepts:
1. Significance level - 5%
2. Confidence level - 95%
3. Statistical power - 80%
4. Confidence interval - depends, e.g., 2%-4%