If you are new, start here:

1. Forecasts Part 1: Choosing the Right Approach - How to predict revenue, user growth, and …

Read more

Subscribe now

Forecasting is one of the most over-discussed topics in data science, and yet it’s still hard to get right. It’s one thing to build a model for Kaggle, but it’s entirely different to deploy it in a real business or product context and fine-tune it while accounting for anomalies, seasonality, external factors, ad spend fluctuations, new feature rollouts, and a hundred other variables we’re expected to quantify

This publication is a follow-up to Forecasting in Analytics: Choosing the Right Approach, published a few weeks ago, where I introduced different types of forecasts and ML models, as well as common use cases for predictive modeling in analytics. I went over common financial and revenue forecasts, including methods like:

1. Historical Growth Rate (or Straight Line)

2. Moving Average

3. Simple Linear Regression

4. Multiple Linear Regression

Today, I take it a step further into data science, covering more complex ML modeling and discussing time series forecasting methods used to predict events, transactions, subscriptions, downloads, and more. I’ll walk you through my approach to forecating, steps, and code.

My approach to forecasting

I typically use 4 different methods (3 ML and 1 projection) to model multiple predictions, ranging from more conservative (lower bound) to more aggressive (upper bound):

1. ARIMA - Moving Average to smooth time series data, w/o adjusting for seasonality.

2. SARIMAX - Moving Average to smooth time series data while incorporating seasonality. This is typically the safest and conservative forecast. Expect it to be on the lower bound.

3. PROPHET - Forecasting for non-linear trends, incorporating seasonality. It's more aggressive in predictions but also proven to be the most advanced among most other forecasts, and often is an accurate method.

4. Projection - Olga’s secret, overly complicated manual projection. I plot every available metric’s historical D/D, W/W, M/M, and Y/Y % change and analyze their (a) correlations and relationships and (b) seasonal thresholds. It takes ages and a kidney to complete, but it consistently delivers the most precise forecast. IF done right. IF I can account for everything the teams are doing.

When reporting, I typically present only Prophet alongside my Projection, keeping ARIMA and its variations for myself as checks.

Prophet forecast

Read more

Subscribe now

I will be in Seattle next week - reach out if you’d like to meet and chat about all things analytics.

Also, this year, I am partnering with Data Council, a “no BS data conference,” and have a 20% discount to share with my subscribers. Use the code daj20 to save on your ticket. The event takes place April 22-24 in Oakland, California. After quite a few years, it’s finally back home in the Bay Area. Hope to see you there.

One of my recent learnings is that it’s surprisingly difficult to find analysts experienced in forecasting and prediction. Which is unexpected, given that regressions and time series forecasting are extensively taught in most academic programs today. But I’ve noticed that analysts tend to either overcomplicate things - applying complex ML to forecast moving average (?) or simply do regression plotting without deeper analysis or understanding the coefficients.

There are many models for forecasting and predictions, each with its own caveats and context. I won’t be going over the basics of forecasts and modeling. Instead, I will focus on the practical side - how forecasting varies depending on the type of project. Forecasting revenue, for example, is very different from forecasting LTV, Churn, or subscription growth, even when using the same model and dataset.

Today, I will cover the different types of forecasts and ML, and the common use cases that require predictive modeling in analytics. I’ll walk you through the steps and models to forecast revenue, ARR, and paid customers to answer questions like:

• How many page views do we need to double signups?

• How many more games, streaks, or logs must users complete before converting to paid?

• How to forecast MRR for the next 2 years.

• When will we reach 1M of subscriptions?

• When will we hit $1M ARR with our current baselines and ad spent?

In another follow-up piece, I’ll share examples of my forecasts and explain how to adapt them to different use cases.

Read more

Subscribe now

Wrapping up the year with another “refresher”. This time, it’s about statistics - compiling all publications on statistics into one concise “take-home” guide.

Applying core statistics in analytics typically involves:

1. Distributions

2. Significance and Confidence

3. Correlations and Regressions

4. Causal inference

5. Law of Large Numbers and Central Limit Theorem

This publication is targeted at analysts, data scientists, and product owners who work with data and ML products.

Why do we need statistics? To bring certainty and confidence

The whole purpose of statistics (statistical theory, methods, and analysis) is to provide certainty out of uncertainty. In other words, when you don’t have a high degree of trust (in either quantity or quality of data), how do you make sure you make the right decision?

Read more

Subscribe now

Welcome back to the Statistics 101 series. Today, let’s talk about:

1. How to get randomly distributed users, using both basic and advanced methods.

2. When to apply sampling techniques to randomize users.

3. Why these techniques matter.

Sampling helps uncover patterns and trends within larger datasets. Samples can be random, uniform, structured, or sorted in specific ways. Here, random does not mean a different number every time, but rather that it can’t be predicted.

When do you need to get randomly distributed users?

Read more

Subscribe now

Today, I want to talk about the most commonly used type of analysis, which is overused by executives, PMs, and engineers yet somehow underused by data scientists and analysts: regressions.

People who borrow data-hat often like to plug numbers into regression models and share plots left and right, using them to back up their hypotheses or assumptions. What they might not realize is that linear regression is a form of ML, which requires data cleaning, feature engineering, addressing underfitting and overfitting, and more. Simply reading scatterplots is not enough. They apply regression without understanding what the scores and equation actually mean or how to interpret them. 

Statisticians, on the other hand, often use more advanced regression techniques and ML models, sometimes making the analysis more complex than necessary.

I am absolutely guilty of the latter.

In this publication, let’s recap:

• Use cases across different regression types.

• Examples of product and marketing hypotheses that can be solved with regression.

• What does the regression equation mean, and how do we interpret the regression scores?

• How to break down scatterplot slope using the regression equation.

• Does regression prove causation?

• How to apply casual inference in analytics.

When and why to use regression

Read more

Beginners often ignore missing values, even when the percentage is impactful. (It’s actually funny - they might report that 86% of the data is missing and then proceed with the analysis or modeling as if nothing is wrong).

Senior data scientists tend to rush to impute missing values with averages, regardless of whether it’s necessary.

There are guidelines for when it’s acceptable to ignore missing data and when it’s not. Obviously, this depends on the volume of missing values. However, it also depends on their distribution, significance, variance, dependence, and the type of modeling you are performing.

Today, I want to recap the most common and, in my opinion, underrated issue in analytics: handling missing values.

Read more

Subscribe now

Thank God for LinkedIn and Paul Levchuk, who enriched us this summer with his research on numerous academic studies on LTV. Now, it’s my turn to pick up the baton.

During my blogging tenure, I’ve tried to stay away from LTV in my newsletter because:

1. LTV is the most over-discussed and over-studied KPI. It feels like everyone is writing about it.

2. There are 100 ways to forecast LTV, 90 of which are very complex. There isn’t one perfect and universally appropriate method to calculate LTV; each has a lot of nuance. 

3. You’re not an analyst until you deliver an LTV prediction. Sooner or later, everyone does it. It’s like graduating into a dark world of under-budgeting and overspending. Fortunately, marketing analytics is not my passion.

4. Most importantly, you don’t need LTV to build a great product and bring it to market. Investors need LTV. Like I said, it’s a dark world of under-budgeting and overspending.

So today, instead of offering my own method for forecasting LTV (and I have 4 main models running, each returning completely different values for the same dataset 🙃), I decided to share 3 case studies on using ML to predict LTV for freemium apps, inspired by and borrowed from Paul Levchuk research.

What I appreciate about Paul is that he’s one of the rare analysts today with strong attention to detail, critical thinking, and thorough due diligence. In other words, you can’t bullshit him with uncontrolled A/B testing, sloppy research, or flawed logic. I enjoy watching him call out inaccuracies in studies, incomplete analyses, or shallow comprehension, whether it’s a junior analyst or an operating partner at OpenView with the largest PLG newsletter.

Read more

Here is what typically data scientists do when working on ML: 

1. Check null values. If they are sparse, remove them. If too many values are missing, find a way to fill them in.

2. Create a distribution of values. Locate outliers. Remove outliers.

3. Convert categorical values into numerical ones for modeling.

4. Group values into features. The more user attributes the dataset has, the better the model performs.

5. The dataset is now clean - there are no outliers, null values, or numerical data - and ready for modeling.

Each of the steps above may be flawed. Some are easier to troubleshoot and improve, while others are more complex and require more context. 

Today, I will focus on outliers.

Outliers are not necessarily bad and do not always have to be removed. It depends on their use case: 

• Certain ML models handle outliers quite well, while others will degrade in performance. 

• While some KPIs and metrics, like DAU, ARR, or Churn, remain unaffected by outliers, others can become misleading, such as Time-to-Value, Transactions Per User, Average actions, etc.

Below, I will discuss the different types of outliers, show how to detect them, and how to figure out when you should remove, keep, or adjust them. Why, in some cases, outliers are harmful, and in others, you have to keep them in your dataset to make your analysis or model more precise and accurate.

Techniques to detect outliers

Read more

Subscribe now

Read more

Subscribe now

Typically, I avoid data engineering and data governance topics in my newsletter. While the success of analytics is directly linked to data governance initiatives, measuring data quality often falls outside the primary responsibilities of data scientists. Also, the subject of data quality is well-covered today, and there is no need for yet another how-to-improve-data-quality publication.

However, a few weeks ago, during the ThoughtSpot webinar on data trends and data quality, a question arose about whether there is a common criteria for defining poor-quality data vs. good-quality data. To my surprise, Sonny Rivera's response was that (a) there is no industry standard to define criteria for data quality, and (b) it has to be “good enough.”

I respectfully disagree with (a), and the only context where I’d settle for “good enough” is when I am preparing my homemade fajitas. 

That being said, I haven’t seen actual KPIs related to data quality. So, I embarked on a quest to find data quality metrics and KPIs I could use to scale the data governance initiatives and measure data quality. Below, after long hours of research, I share what industry leaders have to offer on this subject and my consolidated list of the top metrics you can use to measure data governance ROI and the state of data quality.

What data quality experts offer

First, I checked (*)SELECT STAR blog, the company that is budgeted to solve your data challenges. With the over-promising title of How to Build a Modern Data Governance Framework, here is what they say:

This is the best measurement of data quality the company, which is solving data governance for $900/month, came up with through their 74 blog articles and videos. I found it hard to believe that a company of that scale and funding couldn’t cover the basics of data governance. So, I took time to listen to their videos, but most of them weren’t helpful.

There was one exception, though. During a live YouTube stream interviewing SELECT STAR CEO Shinji Kim, a question arose about measuring the impact of data governance. The host, George Firican, brought up several metrics, including: 

• time spent searching data 

• time spent handling data quality 

• cost-per-decision

• the impact of data governance on some business-related metrics 

Shinji mentioned the time to get compliant, the number of deprecated views and reports, the number of tickets the data team received, the number of solved tickets, and how many times the system(s) goes down because of data changes. However, these metrics primarily measure data team performance rather than the impact of data quality.

Like many of you, when I think of data quality, I immediately think of Chad Sanderson. He developed the concept of data as a product, introduced data contracts, founded a data quality camp community, and eventually co-founded a company solving data quality issues. For years, Chad was publishing and advocating for the importance of data quality. I’ve reviewed , read his guides, watched several of his talks, and am fascinated by the legacy he has created and his detailed approach to understanding and scaling data quality. Yet, I couldn’t find a consolidated high-level list of data quality metrics. I must be a bad detective.

Ergest Xheblati frequently discussed metrics, so I also went through his publications in his . I learned a lot of good insights and bookmarked a bunch. However, none of them focused on data quality metrics. Ergest often mentions SOMA in his publications (I encourage you to check this list of metrics definitions). While it’s really awesome, it primarily consists of business metrics and KPIs rather than indicators for measuring data quality.

Fivetran’s blog, as the “one platform for all your data movement,” referred me to articles on AI and database replication, which, unfortunately, were not particularly relevant. Data Governance vs. Self-serve was an entertaining read.

Perhaps Sonny was right, and there is no industry standard to define criteria for data quality. 

Kevin Hu, CEO at Metaplane, the data observability platform, offers a guide - Data Quality Metrics for Data Warehouses (or: KPIs for KPIs): 

He takes a step further by offering extrinsic vs. intrinsic data quality measurements, which, to me, seemed more complicated than necessary. The article discusses data quality dimensions and their importance, yet it still lacks specific metrics I could utilize. 

Monte Carlo’s Guide on 12 Data Quality Metrics That ACTUALLY Matter presents metrics such as the number of data incidents, time-to-detection, time-to-resolution, table uptime, importance score, table health, table coverage, custom monitors created, number of unused tables and dashboards, deteriorating queries, and status update rate.

How And Why We Need To Implement Data Quality Now! by serves as a good introduction to data governance. He breaks down data quality into 6 measurable pillars, offering examples and use cases for each:

I am going to adopt his approach in creating my list of top metrics for measuring data governance.

How To Measure Data Quality

I noticed that data observability platforms tend to over-complicate the whole concept of data governance with confusing formulas, or introducing weird data ROI Pyramids, or developing data Maturity Curves.

As entertaining as it might be, I want to make the framework for measurement simple and clear, so I decided to condense multiple data quality pillars into only 3 sections with the following KPIs: 

Data Accuracy and Integrity

Accuracy ensures that your data is error-free, precise, and has the right data structure, form, size, or range. It involves checking for errors, duplications, and anomalies.

Metrics can be used to measure accuracy and integrity:

1. Number of identified data discrepancies and inconsistencies

2. % of records containing anomalies.

3. % events with missing attributes or properties.

4. Number of missing rows or columns in a dataset.

5. % duplicated records.

6. Daily AVG of missing values.

7. Volume data/events ready for consumption.

8. Volume data/events used by consumer teams.

9. % incorrectly inserted data.

10. % data delayed or failed to load.

Data Consistency and Completeness

You measure completeness by comparing data against trusted sources. It represents the uniformity in data formats, labels, and definitions across many data sources. Your DAU in Mixpanel should have the same value as your DAU in Postgres, or Signups in GA should match with Signups in Snowflake.

Metrics can be used to measure:

1. % cross-platform unique user variance

2. Total number of transactions (or events) loaded.

3. Total number of transactions (or events) processed.

4. Number of tests implemented / Airflow tasks / dbt tests.

5. Number of audits completed or alerts created

6. Number of data checks monitored.

7. Number of successfully executed jobs in the pipeline

8. Number of business KPIs affected by data anomalies

9. Number of dashboards affected by data issues.

10. % of user IDs obfuscated.

Data Timeliness and Freshness

Timeliness is about data freshness, meaning having data when you need it. Ensure that your data is not only accurate but also up-to-date for relevant and timely insights.

Metrics can be used to measure timeliness:

1. AVG time (min/hours) for data collection.

2. AVG time for data processing.

3. Frequency of data refreshes.

4. AVG time for error detection

5. AVG time for error resolution

6. AVG query execution run time

I am not a data engineer (thank god), so it’s likely to be an incomplete list with missing important measurements. But at least it’s something I can now pass to the Head of Data to measure and scale the teamwork.

I am also frustrated that data experts who shape and influence modern data stacks today publish books, start companies, grow communities, and have enormous reach, and they don’t have these basics and essentials ready and available to borrow. (Most likely, they do, and it’s an issue of discoverability. That is partially a reason why we have to repeat each other over and over again.) 

Thanks for reading, everyone. Until next Wednesday!

1. What Statistics Are Used In Data Analysis? - An introduction to the top four statistical concepts we apply and rely on in analytics.

2. Statistics 101: When Simple Becomes Tricky - Descriptive statistics basics (and a reminder of how easily simple mistakes can slip in.)

3. Applying Statistics In Product Analytics - A deep dive into distributions, their types, and use cases.

Now that you know that most of the analytical world is built on distributions, I will walk you through the cases and examples when it’s acceptable to use Mean vs Median and when it’s okay to use both.   

Central Tendency Theory recap 

If you skipped statistics back in school

There is a lot written out there on the Central Tendency. Here is what you need to know when working with descriptive statistics:

• Central Tendency is a theory about the center of the data set. Why is it important? Because once you locate the center of the dataset, then you can describe it, graph it, and model it. If your center is off or incorrectly defined (e.g., using the flawed Mean), your whole distribution will be off, leading to the wrong outputs affecting your analysis, model, A/B test, etc. In statistics, the truth (and confidence) starts with the center.  

• Mean, Mode, and Median are the top 3 measures of Central Tendency. 

• There are more Central Tendency measures (e.g., weighted average, geometric average, midrange, trimean, root mean square, simplicial depth, etc.). For analytics, most of your work will be around the top 3: Mean, Mode, and Median.

• You would use the top 3 to (a) describe the dataset (hence why, it’s called “descriptive” statistics) and (b) locate the center of the dataset.

• Once you know the center in the dataset, you can locate the dataset ranges, variance, and spread.   

Median vs Mean

Choosing Mean vs. Median mostly depends on two factors: 

1. The data type you work with and 

2. The data distribution.

When you can use both Mean and Median:

• When data is normally distributed (or close to normal distribution), the values are evenly distributed around a central value.

• When the data is symmetrically distributed.

• When the data is continuous or discrete:

  • Discrete: (1, 2, 3, 4…)

  • Continuous: (1.1, 2.45, 3.543…), 26.5%, 30C, 65F, 12.3 miles, 2.5 hours, 10/3/2023…  

As you already know, the Mean is the most commonly used summary statistic. It is essentially a mini model of your data set.

When not to use Mean:

Read more

Subscribe now

Earlier this year, I published Applying ML in Product Analytics, where I covered the types of ML models and their use cases we apply to predict user behavior. 

Today, I want to zoom out from ML and talk about statistical methods we use to describe user behavior. Obviously, first, you have to describe and learn events before you can infer and predict them. 

Applying core statistics in product analytics mostly comes down to understanding: 

1. Causal inference 

2. Distributions 

3. Significance and Confidence

4. Correlations and Regressions

5. Law of Large Numbers and Central Limit Theorem 

This week’s publication is a continuation of my earlier What Statistics Are Used In Data Analysis? and is also a deep dive into distributions aimed at analysts, data scientists, and product owners working with data and ML products.

Below, I’ll share a guide to help you decide which distribution to apply for your use case, analysis, or forecast.

I have a secret fascination with distributions. They circulate all around us in decision-making, analysis, coincidences, probabilities, chances, estimations, forecasts, and more. I believe that reading distributions is a way to understand such phenomena as luck, success, failure, or karma. 

Distributions are so underrated in analytics, and it feels like many schools are on a mission to present distributions as tedious, complex, and unnecessary. Most students don’t quite understand why they need to learn distributions or what to do with them. While researching this publication, I went through dozens of publications, videos, and tutorials on distributions to find any guidance on the “applied” use cases and examples. There is a lot of information about what distribution is and how to understand it, but not much on when/why you would apply which distribution type.

Below is my simplified and practical guide, condensed to only essential must-know concepts for you to understand types of distributions and when to apply them for which analysis or inference. 

When to use distributions

Read more

Now you do! 

You can’t calculate the median of an unsorted dataset (but you can get the mean and the mode). Welcome to another “let me remind you of some basics” data science and analysis newsletter.

We are so accustomed to built-in functions in SQL and Python. I don’t even remember what variance means anymore. I just type VAR() and don’t overthink it. A few weeks ago, I was asked to help replicate STDDEV() step by step and let me tell you, I had to take a good minute to collect my thoughts. 

Very often, you can be asked to break down common built-in functions during technical interviews, e.g. imagine there is no mode() function - write code to get the mode (rumors say the exercises below were asked at Reddit, GitHub, Amazon, and YouTube data science interviews). They are not meant to be difficult but instead aim to test your understanding of statistics, and if you’re able to recognize when and which function to apply.

There is no excuse for failing to replicate the mean or median.  

I assume everyone is familiar with MEAN(), MIN(), and MAX(). I’ll focus below on the basic must-know functions, which you definitely have to use for A/B test analysis, time series analysis, probability estimations, and predictions, such as variance, standard deviation, mode, median, and distribution.

Read more

Subscribe now

Read more

Today’s issue is the final publication in my regression series and probably one of the most important topics in analytics: proving causation.  

Over the last two weeks, I shared an introduction to correlation and regressions, where I covered: 

• How to do linear regression and correlation analysis - what a regression analysis is, when and how to run it, how linear regression is different from correlation analysis, and use cases for each.

• Decoding Regression Scores - what the different types of regression are, when to run which regression method, and how to read regression scores and an equation. 

You already know that correlation does not prove causation. Regression analysis exists in a gray area between cause and effect.

I was taught back at school, that regression may be used to attempt to estimate causal relationships from observational data. And you will find quite a few examples shared by statisticians and researchers where linear regression is used for causal inference.

And yet, if you take an economic theory or MBA, you will learn that regression doesn’t prove cause-effect, and the relationship you see on the linear regression graph does not imply causation.

🤔 So wait, does regression prove or not prove causation?

I like Paul Allison's, Ph.D., professor of statistical methods, take on this that “regression can be used for both causal inference and prediction“ but it all comes down to “how the methodology is used”, or if it should be used at all for a particular problem or question.

Bringing this into applied data science and analytics, any type of regression (like any type of ML) on its own doesn’t prove causation. However, it can be used to get clues and higher confidence that there’s a strong connection between variables, and how this connection will change if we increase or decrease the input values. That being said, if used wrong, and error scores are ignored, it can improperly guide you further away from the truth by misrepresenting data and failing to return the true relationship pattern. 

In this publication, I’ll share some examples of causation analysis and show you ways how not to get tricked by flawed regression output, and how to recognize when the regression pattern you see is correct, trusted, and causal.

Read more

Subscribe now

Last week I shared my recent publication in newsletter about correlation analysis and linear regression. It was more of an introductory piece to the regression, and there is still so much more to cover on it.

So today, it is Linear Regression Part 2, dedicated to my great analysts who are expected to work with ML and effectively understand the concept of regressions. 

Linear regression is the most commonly used analysis to locate the relationship between values and run predictions. It has been widely adopted across product, growth, business, finance, and marketing by all levels of professionals. 

What many people might not realize is that linear regression is a machine-learning model, so you have to treat it accordingly (considering data cleaning, feature engineering, addressing underfitting and overfitting, all of that). Reading its scatterplots is not conclusive. Many analysts apply regression without understanding what the scores and equation actually mean or how to read them. Let’s fix that today.

In this publication, I focus on:

• Breaking down use cases across different regression types.

• Examples of product and marketing questions and hypotheses that you can solve with regression analysis.

• Explain the linear regression equation, what it means, and how to interpret the regression scores.

• How to break down scatterplot slope using the regression equation.

Introduction to regressions and their use cases

Once again, regression shows how much one variable affects another and whether you can use the pattern of one variable to predict and estimate the behavior of another. Any type of regression doesn’t prove causation but rather gives us more clues and higher confidence that there’s a strong connection between variables, and how this connection will change if we increase or decrease variables. 

Possible use cases for regression:

• How many page views do we need to improve signups by at least twice?

• Is showing more recommendations positively affecting trials?

• How many more games/trials/clicks do users have to make to convert to Paid?

• If we send 3x more notifications, how much will this increase DAU?

Read more

“You don’t have to know the code, but you do need to know what the code can do.”

from Stanford DS School 

There are over 50 types of machine learning algorithms with different levels of complexity, similarity, and application. 

How many of these 50 models will you realistically be developing for your projects? 

In fact, mostly two: Linear Regression and Random Forest. Joking! …Or am I?    

With so many algorithms out there, it can be challenging to figure out which model to choose for your analysis (or if you even need to use one at all). This week’s publication is a continuation of What Statistics Are Used In Data Analysis?, and is also an introduction of sorts to Machine Learning 101 (but aimed at analysts).

In today’s publication, I’ll share a guide to help you decide which ML model to pick to solve a problem, and how deep into the woods you’ll need to go with statistics to create a customer churn prediction model or forecast subscription revenue. 

To start with, here’s a quick recap of the main types of ML algorithms: 

Supervised learning - You know how to classify the input data and the type of behavior you want to predict, but you need the algorithm to calculate it for you on new data. The data you work with is labeled, tagged, and has all the inputs and outputs. If you are an analyst, 98% of your work will stay here.

Unsupervised learning - You do not know how to classify the data, and you want the algorithm to find patterns and classify the data for you. A lot of things are happening here. Algorithms process a large volume of data and organize it into clusters (mostly) or some type of structure. The more data you pipe in, the more precise and refined the model will be. If you are a lucky analyst, 2% of your work will be in unsupervised learning, mostly around clusters and recommender systems.

Semi-supervised learning - The line between unsupervised and supervised learning is blurry. Often you have to work with a mix of labeled and unlabeled data. The algorithms use available tags to learn and then mark unlabeled data. This is for cases where you have to use all available data, not only labeled. This also includes even partially labeled data, and you have to make it work for the whole volume.

Reinforcement learning - You don’t have a lot of training data; you cannot clearly define the ideal end state, or the only way to learn about the environment is to interact with it. These algorithms work as an ecosystem that is built on a “trial and error” approach. They learn from errors in already-processed data, adapt, and look for the best result over and over again. 

Using Supervised ML for data analysis

Because it’s all about analytics here, I’ll try to list the most common models used in data analytics and layout use cases for each. 

There are 4 main outputs and the type of analysis we are expected to work with:

1. Regression 

2. Classification 

3. Clustering (unsupervised)

Read more

Analysts don’t work with advanced statistics (unlike data scientists) and are unlikely to deal with complex distributions like exponential, Weibull, or Beta:

And that’s the biggest downside of most statistical classes and tutorials out there. Without the context and foundation, your search for the right solution will take ages and will pull you far off of your original path.

Applying core statistics to real-world problems mostly comes down to understanding casual inference, distributions, significance, correlations, and experimentations. In this issue, I will summarize and cover must-know statistical concepts and their use cases that data analysis couldn’t happen without. 

Statistics 101 

The whole purpose of statistics (statistical theory, methods, and analysis) is to provide certainty out of uncertainty. In other words, when you don’t have a high degree of trust (in either quantity or quality of data), how can you make sure you can make the right decision? 

There are descriptive statistics and inferential statistics. Both are very different approaches to solving and describing the same problems. 

We apply descriptive statistics for big data analytics working with sums, medians, skewed data, variance, and deviations. It’s called “descriptive” because it’s meant to describe the data. It’s used to understand data trends. Every time you have to plot a distribution, you are working with descriptive stats.

Inferential statistics are used to make predictions or infer data trends and patterns. In a nutshell, it allows you to take a sample of data and apply it to a larger population or run hypothesis testing. Every time you work with A/B tests, modeling, or any type of hypothesis or forecast, you are dealing with inferential stats.

Must-know statistical concepts for data analysis

Here are statistical concepts from both descriptive and inferential that you need to know and be able to use comfortably to be proficient at analytics: 

1. Estimations and significance

Read more

If you’re not a paid subscriber, here’s what you missed this month:

• How we optimized the onboarding funnel by 220% - a recap and deep dive into a complete redesign of a user onboarding flow with learnings, data benchmarks, and recommendations. 

• How To Get Paid Subscriptions In SQL - advanced SQL for extracting and computing missing data to report on premium and growth KPIs. If you work in SaaS or B2B and deal with incomplete transactions/subscription data, this approach will save you many hours.  

• Engagement and Retention, Part 4: How To Visualize and Read Cohorted Retention - a continuation of the User Engagement and Retention series with a focus on cohorted retention reporting. This time I offer multiple SQL solutions for calculating retention for different business types, visualizing it, and reading different retention charts. 

Subscribe now

Today I want to do a recap on the most common and, I believe, underrated issue of handling missing values in data analytics and data science. 

If your dataset is sparse or incomplete, you already know about 2 main ways to deal with missing values: you either ignore them or input averages instead of N/As and proceed like nothing bad had ever happened. You may also use more powerful methods, e.g forecasting the missing values or applying ML algorithms. In this issue, I want to offer more of a structured approach that will help you recognize situations where it’s okay to ignore N/As for your analysis or when you should do more thorough data cleaning.

*note: this is all assuming that you can’t obtain or acquire the missing data from other sources. If you can, you should do that first and skip this article. 

Your approach to handling missing values depends on the following questions:

1. What are you working on? (e.g. multivariate analysis, regression, ML) 

2. How much data is missing? (e.g. only a few values, critical values, most values)

3. What is the pattern of missing data? (random, partially random, in pairs, etc)

Ignore or drop missing values:

• If your data is missing at random, you can remove NULLs. 

• If your analysis is multivariate (data containing more than two variables), and if there is a larger number of missing values, then it might be better to drop those rather than do imputation. If the variance is not a factor, you can do it either way. 

Deletion can be done listwise (rows containing missing variables are deleted):

Or pairwise (only the missing observations are deleted):

From How to treat missing values in your data

As a rule of thumb, if missing values are random, you can do a T-test for two data partitions: one with missing values and another one without missing values, and check the difference between the 2 means. If there’s no difference, then ignore the NULLs.

If missing values are not random, you are dealing with a more complicated case. Your data might be still randomly distributed within one or more sub-samples. You can also perform regression or nearest neighbor imputation on the column to predict the missing values (I’ll cover more on this in one of my next issues).

Can’t drop missing values, have to impute: 

• If your analysis is not multivariate, and the values are missing at random, imputation is a good choice, as it will decrease the amount of bias in the data. 

• If the values are not missing at random, you need to do data imputation. 

Check this guide to set up the approach for handling missing values in your analysis:

Handling missing values in Python 

(pandas, dataframe as df)

df.isnull().sum().sort_values(ascending = False)

If you have to fill in missing values: 

df['column']=df['column'].fillna('1') where 1 is the value you want to input

Handling missing values in SQL

In SQL you can leverage DateTime and window functions, and GROUP BY to locate missing values or other data abnormalities. If you have to re-write or input missing values in SQL, you can do it in 2 ways: 

1. During the schema design step, you can set NULL values to 1, for example, or drop the column if needed, rename, or set its parameters.

2. If you create VIEW, a temporary table with merged datasets, or CTE, you can use a CASE expression to rename or re-define some values.

3. In your SELECT statement, you can use a CASE expression for data input the same way. 

4. You can use ALTER to run transformations for the already created tables. This probably won’t work if you are dealing with a huge volume of data. Keep in mind that ALTER can be costly. 

You can check more Python methods on how to change a data format, input values, and rename fields in my analysis, or in this guide - PostgreSQL vs Python for data cleaning: A guide. 

Thanks for reading, everyone!