Subscribe now

Date transformations are annoying and time-consuming, regardless of whether it’s SQL, Python, R, or English. I often get stuck on the most basic things, like dates. It is very frustrating.

I want to share this reference guide today, mostly so I can have it close for myself. I still have to use it at least once per week. Hopefully, it will save you time when working with UTC-to-datetime conversions, parsing strings to datetime, or handling anything dates-related.   

For Python Pandas, I’ve compiled many (if not all) known variations of datetimes solutions for any case, which you can see below. If I missed something common (I am sure I did), please let me know, and I’ll add it. 

Read more

If you are in a leadership position, it’s your responsibility to ensure that you will be able to contain, own, and ideally prevent whatever errors your team makes.

Today I’ll grudgingly share some painful lessons I learned from my own errors made while working with SQL and Python to extract, read, and analyze data. These mistakes are so simple and basic, which makes it that much harder to accept. I hope it will remind us of the importance of triple-checking, code reviews, and cross-team validation. 

Read more

Throughout the year, I covered most steps of data work in Python including its packages, data ingestion, cleaning, EDA, graphs, and more. One important piece of workflow in data science and analytics that I haven’t touched yet is data processing. Specifically: merging multiple datasets in Python Pandas. 

This step was somehow the hardest for me to figure out, and I had my share of mistakes made using the wrong approach that delayed my analysis (or even worse, brought me to the wrong output leading to getting the wrong data). 

This step is the most tricky one because it’s done early in the process and sets the baseline for your analysis. What this means is if you initially merged the datasets wrong, every next step will get you further from the truth. In this issue, I’ll walk you through the methods of merging multiple datasets into one and describe the difference between MERGE(), JOIN(), and CONCAT() in Python, and give you pointers to follow to figure out which approach you should use for which case. 

Python Pandas is particularly great for any use cases in data analysis. I can see how it can be daunting to search for documentation to figure out what is the right or best way to perform a particular task, especially when you don’t know what you’re searching for. While I encourage you to start with reading documentation, I also hope to point you to the right method and save you some time. 

Read more

The assignment and my solution contain ex…

Read more

Plots, plots, plots! Hate them or like them (I know the expression is usually “love”, but let’s not get ahead of ourselves), you can’t deliver a thing in analytics without them. 

This publication is a follow-up piece to my recent How To Pick The Right Chart article where I covered the most common charts and offered guides on how to appropriately choose the right visualization for your analysis.

I am also publishing this reference guide mostly for myself, in order to have a consolidated list of the most useful plot types and plots tutorials. There is a lot of documentation on Python plots (check some at the end of this newsletter) but this is meant to serve as a sweet one-pager that you can quickly refer to while picking a plot for your analysis. 

Read more

That’s why I created this reference guide. It will save you time when working with the code. 

I’ve created over a dozen of them for every language or any type of case. In my journal, I’ve published only a few so far but will be publishing more. Today, it’s all about DateTime transformations in Python Pandas - a compilation of tips, solutions, and workarounds for any possible case with DateTime formatting.   

Subscribe now

If you work with SQL, make sure to check out SQL Date-Time Guide for a list of the most common DATE extractions solutions. 

For Python Pandas, I’ve compiled many (if not all) known variations of DateTimes solutions for any case, which you can see below. If I missed something common (I am sure I did), please let me know and I’ll add it. The tips and solutions below are taken throughout the web from multiple sources, including 100 pandas tricks created by Kevin Markham and just-pandas-things. You can copy the code below or fork and run it from this Kaggle notebook.

df = generate_sample_data_datetime().reset_index()

df = df.sample(500)

df["Year"] = df["index"].dt.year

df["Month"] = df["index"].dt.month

df["Day"] = df["index"].dt.day

df["Hour"] = df["index"].dt.hour

df["Minute"] = df["index"].dt.minute

df["Second"] = df["index"].dt.second

df["Nanosecond"] = df["index"].dt.nanosecond

df["Date"] = df["index"].dt.date

df["Time"] = df["index"].dt.time

df["Time_Time_Zone"] = df["index"].dt.timetz

df["Day_Of_Year"] = df["index"].dt.dayofyear

df["Week_Of_Year"] = df["index"].dt.weekofyear

df["Week"] = df["index"].dt.week

df["Day_Of_week"] = df["index"].dt.dayofweek

df["Week_Day"] = df["index"].dt.weekday

df["Week_Day_Name"] = df["index"].dt.weekday_name

df["Quarter"] = df["index"].dt.quarter

df["Days_In_Month"] = df["index"].dt.days_in_month

df["Is_Month_Start"] = df["index"].dt.is_month_start

df["Is_Month_End"] = df["index"].dt.is_month_end

df["Is_Quarter_Start"] = df["index"].dt.is_quarter_start

df["Is_Quarter_End"] = df["index"].dt.is_quarter_end

df["Is_Leap_Year"] = df["index"].dt.is_leap_year

Convert year and day of the year into a single DateTime column

d = {\
"year": [2019, 2019, 2020],
"day_of_year": [350, 365, 1]
}

df = pd.DataFrame(d)
df

# Step 1: create a combined column

df["combined"] = df["year"]*1000 + df["day_of_year"]
df

# Step 2: convert to datetime

df["date"] = pd.to_datetime(df["combined"], format = "%Y%j")
df

Convert from UTC to another timezone

s = pd.Series(range(1552194000, 1552212001, 3600))

s = pd.to_datetime(s, unit = "s")

# set timezome to current time zone (UTC)

s = s.dt.tz_localize("UTC")

# set timezome to another time zone (Chicago)

s = s.dt.tz_convert("America/Chicago")

Related publications: 

• A Selection Of Python Tutorials for Analysts

• How To Install And Set Up Python

• Correlation Analysis 101 in Python

• SQL vs Python For Data Cleaning

• How To Pass A Take-Home Python Assignment

• Python Questions For A Data Analyst Interview

Thank you for reading. Until next Wednesday!

Read more

Subscribe now

A little pre-history: I started my analytical journey with R (like any gray-haired medieval analyst today). And after all that time, I still think R language is the most suited for statistics and analysis. I had to make a switch because every team I was a part of was using Python, and many projects we collaborated on were also done in Python. 

My transition from R to Python was … long. I think I went through every Python class out there. After too many tutorials, I started making a list of my favorite go-to videos and lessons which I still use and keep updating to this day. Most of those are (or were) free, but things change, so it is possible some of the classes below are not free anymore.  

You probably have noticed already that I’m a fan of Real Python tutorials. I think they have a good combination of theory and examples, covering everything from basics to advanced development. So many of my favorite sources come from there.

Installation

• Python Basics: Setting Up Python

• Installing Python on Windows, macOS, and Linux

• Setting up a Python Development Environment in Sublime Text (if anyone is using Sublime)

• Python Tutorial: Anaconda - Installation

Basics

• How To Import Data Into Python?

• Reading and Writing CSV Files

• Reading and Writing Files With Pandas

• Using Jupyter Notebooks

• The Pandas DataFrame: Working With Data Efficiently

• Pandas, Data Frame, and Data Series - really like this channel. Data frame and series explanations are very good.

• Pandas DataFrames 101

Data Types

• Basic Data Types in Python

• Python cheatsheet - basics

• Data Structures and Algorithms in Python - Full Course for Beginners

• Variables in Python

• Lists and Tuples in Python

• Python Tuple

• Python Lists - Basics

• Dictionaries and Arrays: Selecting the Ideal Data Structure

• Python Dictionaries

• Defining and Calling Python Functions

Data cleaning and EDA

• Data Preprocessing with NumPy

• Pythonic Data Cleaning With Pandas and NumPy

• Pandas Sort: Your Guide to Sorting Data in Python

• Pandas GroupBy: Your Guide to Grouping Data in Python

• Grouping and Aggregating - Analyzing and Exploring Your Data

• Filtering - Using Conditionals to Filter Rows and Columns

• Cheat Sheet for Exploratory Data Analysis in Python - infographic by Analytics Vidhya

• Explore Your Dataset With Pandas

• Data Analysis with Python (EDA)

• Cleaning Data - Casting Datatypes and Handling Missing Values

• Working with Dates and Time Series Data

• Generating Random Data in Python (not cleaning or EDA, but helpful for testing)

Analysis and some simple ML

• Python Statistics Fundamentals: How to Describe Your Data

• Introduction - Data Analysis and Data Science with Python and Pandas

• NumPy, SciPy, and Pandas: Correlation With Python

• Seaborn Tutorial

• Linear Regression in Python

• Linear Regression

• Logistic Regression in Python

• Splitting Datasets - train_test_split()

• Intro into NLP

• Machine Learning Tutorial with Python Intro

• Creating an SVM from scratch

SQL

• Introduction to Python SQL Libraries

• SQLite Databases With Python - Full Course

• Practical SQL for Data Analysis

Plots

• Plot With Pandas: Python Data Visualization Basics

• Matplotlib – An Intro to Creating Graphs with Python

• Creating and Customizing Our First Plots

• Python Plotting With Matplotlib

• Chart Visualization - pandas documentation

• Python Histogram Plotting: NumPy, Matplotlib, Pandas & Seaborn

• Bar Charts

• Stack Plots

• Plotting Time Series Data

• Scatter Plots

• Histograms

Other

• Learn Python - Full Course for Beginners - a full complete course (4 hours) of Python Intro.

• Python Data Science Handbook - The entire Python Data Science Handbook, in the form of free Jupyter notebooks.

• Astronomical Data in Python - the code is written in Jupyter notebooks. You can run the notebooks either on Colab or in your own environment (you can download them from the repository and follow the instructions to set up your environment).

• StatQuest with Josh Starmer - less Python and mostly ML.

• Ultimate Python study guide - all in one: a good resource and guide. 

• Python from Nisha M - a good explanation about data types.

I have another list for SQL and Statistics tutorials, which I’ll share soon as well. I continually update this list to keep it fresh and relevant, so please, send me your favorite Python links and sources that you think I should add!  

Thanks for reading, everyone. Until next Wednesday!

Share

Did they ask you not to spend more than 2-4 hours on a challenge? Haha. Very funny. But this is data analysis, not a comedy club. Let’s get down to business!

Let me do a proper intro. This publication is a successful example of a take-home assignment for …

Read more

There are multiple ways to solve this challenge, ranging from easy workarounds for one-time ad-hoc requests or writing scripts that would allow automating revenue reporting. Depending on your applications, you could simply leverage the window function, utilize the built-in currency converter, or request support from your engineering team to run this conversion on a backend. If none of these are an option for you, below I’ll demonstrate an easy and quick workaround as a way to get it done on your own using only SQL. 

Problem

Let’s say you have all purchases loaded in a table with user id, user country, and main purchase details like plan id, plan name, amount, currency, and transaction date. You have to use this table to calculate MRR per country per period or get total revenue made internationally:

If your desired currency is dollars, the US revenue is easy to report. In your SELECT statement, you would just do sumSUMamount_paid). But what about International? 

You can’t simply pick the expected amount per plan and report revenue based on it, because the plan amount is different from the amount paid. The amount paid is actual money received from users, including discounts, promo codes, refunds, etc. That’s why you have to convert every paid amount into dollars or the baseline reporting currency at your company. If only 5 or 10 countries are supported, the conversion rate can be calculated manually in SQL. However, if you deal with many countries, then you’d better pick another approach.  

Solution

The obvious solution to this problem is to create a helper table with exchange rates per currency. Below I offer you 2 solutions on how to generate this table: 

1. We leverage public rates of exchange data and load them into a view or a table with a primary key on currency code, and then simply perform a join to map the currency we have to the appropriate conversion rate to calculate a dollar amount for every transaction. 

This solution will work for every database and SQL variation. It’s simple, easy, and fast. The downside is that it’s a static table that won’t work as a long-term solution, because the currency rates fluctuate. When you report revenue data, you obviously should be as precise as you can. But for ad-hoc or one-time data requests, the above is the best way to proceed. 

2. We use a Python script to access public exchange rates data via API, and then transform JSON files into a SQL INSERT statement and load it into a generated table.

This is a more complex approach similar to ETL. It gives you refreshed, current exchange rates and keeps the table updated for any long-term or automated reporting. 

There are multiple ways to achieve this. My workable approach was to leverage sqlalchemy and SQLite, and the destination table can be loaded right into Snowflake, Postgres, or any database.

Check Python code here.

Thanks for reading, everyone. Until next Wednesday!

Read more

Read more

Do you ever receive questions like: 

• Does correlation imply causation?

• How do you prove if features X and Y are correlated?

After this helpful guide, you will know the best way to answer those types of questions. In this article, I’ll focus on positive and negative correlation analysis and specifically cover: 

1. Practical use cases for correlation analysis.

2. A methodology for how to build a correlation table and a heatmap in Python Pandas.

3. How to read and interpret different heatmaps and correlation charts. 

4. How to prove that correlation implies causation.  

   

   Subscribe now

When do you need to run a correlation analysis?

From a business perspective, correlation analysis helps you to answer questions like: 

1. What is the relationship between 2 features on your product app? 

2. Are they dependent or independent? Do they increase and decrease together (positive correlation)?

3. Does one of them increase when the other one decreases and vice versa (negative correlation)? Or are they not correlated?

4. Does changing a price affect subscription creation? 

5. Does an increase in comments affect reshares? 

There are multiple other analytical techniques that can help you tackle those or similar questions like hypothesis testing, decision trees, network analysis, matrix, or sorting. Correlation analysis is one of the more common ways to learn the relationship between 2 or more variables. 

Correlation is represented as a value between -1 and +1 where +1 denotes the highest positive correlation, -1 denotes the highest negative correlation, and 0 denotes that there is no correlation. Below, I’ll demonstrate how to run correlation analysis using Python Pandas and read a heatmap. 

How to build correlation analysis

Building a correlation chart in Python Pandas is very easy. 

First, you have to prepare your data by having only numerical and boolean variables in columns (other formats will be ignored by the function). You don’t have to worry about missing/NULL values here, as the function excludes them. After that, you can simply run:

DataFrame.corr()

or

DataFrame.corr(method ='pearson')

This is for a DataFrame. You also can run Series.corr() to compute the correlation between 2 series.  

DataFrame.corr() returns a correlation table between dataset variables. Here is an example of output from Reddit Exploratory Data Analysis in Python:

We see that both the score and number of comments are highly positively correlated with a correlation value of 0.63. There is some positive correlation of 0.2 between total awards received and score (0.2) and num_comments (0.1).

💡Note: this works for the Pearson correlation type, which is the most commonly used standard correlation. There are also Kendall and Spearman rank correlation types. You can specify the method of your correlation like this: DataFrame.corr(method ='kendall'). To learn the difference between these methods, I recommend reading this guide. But in a nutshell, we use Pearson to find a linear relationship between normally distributed variables. When the variables are not normally distributed or the relationship between the variables is not linear, we would use the Spearman rank correlation method.  

Now let's visualize the correlation table above using a heatmap:

It returns a correlation heatmap plot:

The full analysis is Correlation Analysis Using Python Pandas. 

How to read correlation charts:

Each square shows the correlation relationship between the variables on each axis. As I said above, correlation ranges from -1 to +1. 

• Values closer to 0 mean that there is no linear trend between 2 variables. 

• The close to 1 the correlation is the more positively correlated they are, the stronger this relationship is. 

• A correlation closer to -1 is similar, but instead of both increasing like the example above, one variable will decrease as the other one increases. 

• The diagonals are all 1 and marked dark because those squares are correlating each variable to itself (so it's a perfect correlation, therefore it is 1). 

Overall, the larger the number and the darker the color, the higher the correlation between 2 variables. The plot is also symmetrical and diagonal because the same 2 variables are being paired together in those squares.

Another example from my EDA analysis - Predicting Titanic Survival. This plot is meant to show if there is a relationship between such variables as having children, parents, siblings, expensive tickets (fare), or specific age, all compared to passenger survival on the Titanic. It might be a grim example, but it’s a good one:

As you noticed, there are no high correlations in this graph. The highest value is 0.4 between parents/children and siblings. This relationship analysis guides us to the right input variables to create a ML model to predict passenger survival.  

How to prove correlation implies causation?

In addition to the correlation table and a heatmap, for some of the business cases you should consider other factors based on historical data, events, user attributes, and business case specifics:

Strength - a relationship is more likely to be causal if the correlation coefficient is large and statistically significant. This is directly related to the correlation table output data.

Consistency - a relationship is more likely to be causal if it can be replicated. 

Temporality - a relationship is more likely to be causal if the effect always occurs after the cause.

Gradient - a relationship is more likely to be causal if greater exposure to the suspected cause leads to a greater effect. This is related to positive or negative correlation. As I stated above, negative correlation occurs when one variable decreases as the other one increases. 

Experiment - a relationship is more likely to be causal if it can be verified experimentally. You can run hypothesis testing to prove it.

Analogy - a relationship is more likely to be causal if there are proven relationships between similar causes and effects.

That’s it for now. Until next Wednesday!

There are many applications, tools, and libraries that can help you to generate a word cloud in mere seconds for free (you can check some of those below). That being said, as an analyst, you should be able to create your own visuals in either R or Python, both of which should grant you the freedom to tailor your dataset as needed. Pick a style and customization that works best for you!

Subscribe now

Word clouds - what and why

First things first! You’ll need to make a word cloud if you want to visualize which words are used the most in your dataset. The more often a word is used, the larger it will appear in your cloud. Text clouds are the best option when you have to quickly find a pattern, insight, or note a frequency of words used in your data. This will be your first request for any Exploratory Data Analysis tasks with text data.

Getting started

For my analysis today, I am choosing the Python wordcloud package. We’ll use NumPy and Pandas for data processing. 

import numpy as np # linear algebra

import pandas as pd # data processing

import seaborn as sns #statist graph package

import matplotlib.pyplot as plt #plot package

import wordcloud #will use for the word cloud plot

from wordcloud import WordCloud, STOPWORDS # optional to filter out the stopwords

You don’t have to use stopwords to generate a word cloud. It’s advised to use them, however, in order to eliminate the text noise. You also can set a list of stop words to anything you like: 

stop_words = set(['have', 'when', 'about', 'according', ‘who’, 'actually','zero', ''])

💡 Tip: if you are unfamiliar with the package and its functions or limitations, you can simply run ?WordCloud to get its documentation.

Prepare dataset

Before we proceed with the cloud, we have to tailor our dataset to ensure the values are in an appropriate format.

First, we have to remove NULL values:

df["title"] = df["title"].fillna(value="")

Now, let’s add a string value instead to make our Series clean:

word_string=" ".join(df['title'].str.lower())

and... Plotting!

plt.figure(figsize=(15,15))

wc = WordCloud(background_color="purple", stopwords = STOPWORDS, max_words=2000, max_font_size= 300,  width=1600, height=800)

wc.generate(word_string)

plt.imshow(wc.recolor( colormap= 'viridis' , random_state=17), interpolation="bilinear")

plt.axis('off')

We set 2000 words limits for this cloud. Let’s try setting the limit to 50 and changing the background color:

plt.figure(figsize=(15,15))

wc = WordCloud(background_color="yellow", stopwords = STOPWORDS, max_words=50, max_font_size= 300,  width=1400, height=800)

wc.generate(word_string)

plt.imshow(wc.recolor( colormap= 'viridis' , random_state=17), interpolation="bilinear")

plt.axis('off')

The full code is on Kaggle - Word Cloud using Python Pandas.

💡You probably noticed I am using imshow. It’s a function from matplot package that transforms your data into an image. To set or change its parameters, follow this guide.

That’s it for now. In one of my next issues, I’ll demonstrate using masks for generating clouds in a form of a star, circle, or any shape that you could possibly ever want!  (Maybe.) 

Check out a list of my favorite go-to online word cloud generators:

1. TagCrowd.com

2. MonkeyLearn.com - you have to create an account, but once you are set, they provide a lot of text semantical analysis.

3. WordArt.com

4. WordItOut.com - works the best with a cleaned text. 

5. WordCloudMaker.com

Thanks for reading, everyone. Until next Wednesday!

Share

Last week I had the pleasure to speak with a UC Berkeley student, Shalini, where we shared our experience, love, and pain of navigating through the sometimes-rough waters of data analysis. Check out her YouTube …

Read more

✨ Today we will be discussing: 

• Snowflake, a data analytics and cloud platform, is the biggest IPO in 2020. 

• Database tunning - what and why.

• How to work with structured and unstructured data.

• Hackathons round up -…

Read more

When you work with Pandas, the most important thing to understand is that there are two main data structures — Series and DataFrame:

• Series is a one-dimensional indexed array that…

Read more