When Simple Becomes Tricky: Making Sense of Averages in Reporting - Issue 229

From Outliers to Denominators: How to Use Metrics That Tell the Real Story

Welcome to the Data Analysis Journal, a weekly newsletter about data science and analytics.

---

This publication is part of my ongoing KPIs Done Wrong series, where I cover common pitfalls, ineffective proxies, and misapplying benchmarks. I am sharing this piece a little early to keep the momentum going on Do’s and Don’ts of metrics reporting.

When developing reporting for KPIs, it is important to recognize there are 4 types of metrics:

1. Sum and count: DAU, the sum of sales, total End-of-period paid subscribers, etc.

2. Distribution (mean, median, mode, percentiles): average memory used, % of MAU doing X, a median session length... 

3. Rate and Probability: click-through rate or click-through probability.

4. Ratio: Monthly/annual subscription ratio, male/female usage ratio, etc.

Rates, Probabilities, and Ratios are often the hardest metrics to automate because they require calculations and multiple variables.

However, Distributions are the most tricky. They may seem simple and common, after all, every analytic tool offers built-in reporting for averages, median, and percentiles. However, you need due diligence and statistical knowledge to make these reports accurate and actionable. 

So today, let’s talk about the often-overlooked complexity of averages and discuss how to develop a system of checks to ensure your reports are accurate and trusted.