How To Locate The Right Frequency Of Push Notifications - Issue 135

An analysis to find the right threshold of the most impactful frequency of user communication

Finding the most optimal user communication frequency is a challenge that analysts and data scientists often willingly accept in order to support marketing and product teams. It can be setting email cadence, push notification frequency, ads impressions, or payment upsells. 

In today’s publication, I’ll share with you methods and analysis on how to find the right threshold of notification frequency to make sure it converts into the highest DAU, while at the same time also doesn’t harm user engagement. This is not a straightforward analysis, but fairly common to tackle growth in product and marketing analytics.

A few weeks ago in my publication about applying ML in product analytics, I shared a guide to help you decide which supervised and semi-supervised ML model to pick to solve a problem. I pointed out that regression is the most common ML we use to answer many product and BI questions including LTV predictions, revenue forecasts, estimating the number of new page views needed to improve signups, how many notifications increase DAU, and so on. 

But:

Linear Regression isn’t the right method of getting the threshold between multiple outputs.

While regression is the right ML to apply in order to understand the relationship between a metric and user actions (and how much change in one variable affects another), this is not the right approach for finding the right threshold for problems with multiple outcomes. For example, measuring a variable (e.g. emails, notifications, ads) against 2 or more output metrics (like DAU, activations, CVRs, unsubscribes, churn, etc) is not a regression task, and the best method here would be to leverage (a) classification models and (b) testing.

Before we dive deep into that, a quick reminder on regression:

• Use a simple regression model with a single independent variable:

  • screen view → activation

  • Diwali campaign impressions → trials

• Use a multiple regression model with 2+ independent variables:

  • screen view + age → activation

  • upsell click + blog opt-in + Octoberfest → churn

Please notice, multiple regression is designed for multiple input values, not output metrics. So, you can’t apply it to estimate clicks and unsubscribes at the same time. And that’s the reason why classification models would fit better for this task than regression. At least, that’s what I initially thought.

But while researching for this publication, I learned that there is also a multi-target regression that, theoretically, fits into our case. So instead, we should:  

• Use a multi-target regression model with multiple dependent variables:

  • screen view → activations + DAU + churn

  • notifications → DAU + unsubscribes

I haven’t used multi-target regression yet. I am going to bravely explore it and report back on how well it fits into our task of finding the right threshold for notifications. Meanwhile, below are my proven methods of analysis to locate the right balance of notification frequency between high CTRs and low opt-outs. 

The context - why do we even need to have a threshold of notification frequency? What are the risks?

Too low frequency of notifications: missed opportunities to sell and to engage. Reduced app visibility. Even if a user doesn’t click on the notification and ignores it, they are still reminded about your product and brand which leaves you the opportunity of a winback.

Too high frequency of notifications: the more notifications users receive, the less likely they are to open them. Creates an increase in opt-outs or, even worse, app uninstalls or deleted accounts. Once the user abandons the product, it’s very difficult to get them back. High uninstalls may plummet your app store ratings.

So push notifications are tricky, and your team has to be smart about setting the cadence and frequency. This is when you come into play.  

Analysis to estimate the right threshold for push notifications

Below I share 3 approaches for such a task: