Why Trial Success Rate Is the Hardest KPI To Accurately Report - Issue 243

A deep dive into the Trial-to-Paid Rate: calculations, methods, challenges, and trusting data from Subscription Analytics apps

Welcome to my Data Analytics Journal, where I write about data science and analytics.

---

To be fair, the Churn Rate and Net New Anything would theoretically be the most difficult KPIs to work with. However, the Trial Success Rate falls into that category as well.

The tricky part is that it seems so simple - you take converted trials and divide them by started trials. That’s it. Yes and no. Analysts often underestimate how much precision is required to define this metric accurately and make it actionable.

In practice, there are different types of trials: one-time eligible trials, infinite freemium, and reverse trials, all of which require customized logic for trial conversion. Also, trial durations can vary - 7 days, 14 days, 30 days, 90 days, or worse - a combination of a few - which makes mapping them into monthly or weekly metrics reporting tricky.

To make matters worse, no existing application provides an accurate Trial Success Rate. I've used Chargebee, Recurly, RevenueCat, ChartMogul, and Baremetrics, and none of them gave me an accurate Trial Success Rate. As Stripe or Apple dashboards. I always had to rebuild it myself.

Below, I’ll walk you through my approach to calculating an accurate Trial-to-Paid Rate and addressing different trial-length conversions in weekly or monthly reports.

Why we can’t have nice things

Or why most SaaS or Subscription apps fail at the TTP Rate

First, some subscription analytics apps do not offer insights into trial usage or trial-to-paid conversion rates because the data they can access comes from the payment stores only after users have verified and billed. As a result, trialers can be “invisible” in some subscriptions or SaaS apps (which was the case for ChartMogul and Chargebee)

This is also a common issue with the payment_success event in Amplitude or Mixpanel when the trial flag is not passed. Even when it is passed, setting up the TTP rate chart via a funnel is often inaccurate.

The biggest and most common challenge with subscription reporting in RevenueCat, Recurly, and similar apps is that they don’t have a concept of a re-subscriber. This means they cannot accurately differentiate between:

1. New subscriptions activated from trials.

2. New subscriptions activated by users who are no longer eligible for a trial.

For example, let’s say a user starts a 7-day trial on April 1st and cancels it on April 4th. The dashboard records a failed conversion on April 4th. Now, imagine the same user returns three months later and subscribes directly to a paid plan on July 1st. Since they’ve already used their trial, they are no longer eligible for another one, so they activate a paid subscription immediately.

Here’s where modern subscription reporting apps go wrong: they incorrectly mark this new subscription as a TTP conversion and tie it to the initial trial start date (April 1st). As a result, on July 1st, the dashboard updates the TTP Rate for April 1st instead of logging a new conversion for July 1st. 🫠

This is a critical issue because the user's return may be the result of a win-back strategy, but there’s no way to measure it.

That’s why most subscription analytics apps inflate TTP rates - they count both successful trial conversions and re-subscribers. This is especially problematic for mature apps, where re-subscribers can account for 20% of new subscriptions, leading to significantly over-reported TTP rates.

This becomes even more complicated when you also start offering different plans or prices. It becomes almost impossible to attribute the correct plan to the right conversion for the appropriate timeline.

How the TTP Rate should be calculated