From Analytics to Data Science: Building Forecasts - Issue 257

A step-by-step guide to ARIMA and Prophet to forecast events, transactions, and customer growth.

Welcome to the Data Analysis Journal, a weekly newsletter about data science and analytics.

Forecasting is one of the most over-discussed topics in data science, and yet it’s still hard to get right. It’s one thing to build a model for Kaggle, but it’s entirely different to deploy it in a real business or product context and fine-tune it while accounting for anomalies, seasonality, external factors, ad spend fluctuations, new feature rollouts, and a hundred other variables we’re expected to quantify

This publication is a follow-up to Forecasting in Analytics: Choosing the Right Approach, published a few weeks ago, where I introduced different types of forecasts and ML models, as well as common use cases for predictive modeling in analytics. I went over common financial and revenue forecasts, including methods like:

1. Historical Growth Rate (or Straight Line)

2. Moving Average

3. Simple Linear Regression

4. Multiple Linear Regression

Today, I take it a step further into data science, covering more complex ML modeling and discussing time series forecasting methods used to predict events, transactions, subscriptions, downloads, and more. I’ll walk you through my approach to forecating, steps, and code.

My approach to forecasting

I typically use 4 different methods (3 ML and 1 projection) to model multiple predictions, ranging from more conservative (lower bound) to more aggressive (upper bound):

1. ARIMA - Moving Average to smooth time series data, w/o adjusting for seasonality.

2. SARIMAX - Moving Average to smooth time series data while incorporating seasonality. This is typically the safest and conservative forecast. Expect it to be on the lower bound.

3. PROPHET - Forecasting for non-linear trends, incorporating seasonality. It's more aggressive in predictions but also proven to be the most advanced among most other forecasts, and often is an accurate method.

4. Projection - Olga’s secret, overly complicated manual projection. I plot every available metric’s historical D/D, W/W, M/M, and Y/Y % change and analyze their (a) correlations and relationships and (b) seasonal thresholds. It takes ages and a kidney to complete, but it consistently delivers the most precise forecast. IF done right. IF I can account for everything the teams are doing.

When reporting, I typically present only Prophet alongside my Projection, keeping ARIMA and its variations for myself as checks.

Prophet forecast