Why Notebooks Are Game-Changers in Analytics - Issue 215
From Jupyter to Deepnote: How notebooks accelerate collaboration, streamline workflow and foster data culture
Happy last day of July, and welcome to my Data Analytics Journal, where I write about data science and analytics.
This month, paid subscribers learned about:
Data Analysis in Python: A Guide to Working With Dates in Pandas - A quick reference guide for handling and parsing dates in Python using Pandas.
What Is the Optimal Free Trial Length? - How long should your trial period be: 7 days, 14 days, or 30 days? Research and case studies.
Why it’s wrong to run A/B test for too long - The risks of extending the A/B test timeline and why duration matters.
I just returned from the French Riviera to the unexpected news that's causing a stir in the analytics community this week: Deepnote acquired Hyperquery!
As a newsletter writer, I suppose I am on duty to offer commentary on what this means for all of us, why this acquisition happened, and whether it will transform the world of notebooks. I also wonder if Omni should buy Hashboard now, if Sigma is better off acquiring Equals, or if Amplitude should merge with Mixpanel for the heatmaps...
I won’t delve into these speculations, but I will take any opportunity to advocate for notebooks. So today, let’s talk about different types of notebooks, why notebooks are great, and how they accelerate analytics, streamline work output, and foster a data culture.
If you have been reading my newsletter, you already know I am a big fan of notebooks. I think I have had a chance to review or test almost every notebook on the market. And I like them all!
Jupyter
For many years I was a fan of Jupyter Notebooks.
The biggest drawback with Jupyter notebooks, I find, is that they work locally, and it’s not easy to add more users to work on the same notebook (although I know there is a way to do that using JupyterHub). For that reason, while I like Jupyter, I switched to Google Colab.
Otherwise, it’s a great tool - free and open-source, easy to use, interactive, offers a good community support, and can be easily embedded on web pages. It supports Python, R, and Julia. If you are getting started with Jupyter Notebooks, start here - the Jupyter Notebook IPython tutorial on Binder.
Google Colab
The only difference between Colab and Jupyter is that Colab is cloud-based and has integrations with the Google Cloud stack. So you don’t need to install it, and there is no need for configurations. You can simply open it from a browser, invite your team, and work together on the code just like you would collaborate in a Google Doc.
In a way, Colab is like Kaggle, but I like it more because (1) it runs way faster, especially if you use TensorFlow, and (2) it lets you store and save work to your Google Drive. It makes it easy to upload to GitHub repositories. I'm not sure if it would work for bigger teams, but Colab is great for side projects.
Hex
Then enter Hex:
My story with Hex started back in the good old days when it was a simple and affordable notebook. I liked it! You could synchronize it with GitHub/GitLab, structure your documentation and projects, and collaborate with multiple analysts.
Today, it has grown into something massive and weird, making it intimidating to bring in for my team. I’ll quote Benn Stancil from our recent interview, as his thoughts really resonate with me:
“It’s a very good notebook. And initially, they were focused on being a notebook tool. But now, they position themselves more as BI, and the notebook is the backend for the BI tool. It’s notebook-based BI, rather than a notebook. This is the drift that makes BI so hard - it’s easy to evolve from building a great, specialized tool into building a much more general BI tool with a twist.”
It seems to me that Hex has become less of a notebook and more of “everything analytics.” The mission is easy to fail at. I really would love to have a simple, good notebook.
Hyperquery
Then, I had a chance to meet Robert Yi 🐳 and Joseph and explore Hyperquery.
It is a simpler, more intuitive Notion-like tool that offers many of the notebook features you need when working with analytics - visuals, quick sheets, and Python+SQL support.
I liked Hyperquery more than Hex, as it felt more intuitive and not overloaded with features and buttons, while still offering the same support for collaboration and distribution.
Deepnote
Last year, I met Jakub and became fascinated with Deepnote. Deepnote is a more mature data science notebook that is also free to get started with. It allows you to collaborate with multiple teams, share comments, create visuals, store and distribute work, and schedule notebook runs with emailed logs, and much more.
Their support for academia, ease of use, and seamless distribution completely won me over. As of today, I am convinced Deepnote is the best notebook on the market, and I have been using it for over a year.
A few months ago, I shared one Data Portfolio featuring a deep-dive analysis completed in Deepnote by my fellow analysts (Here is their complete analysis Data Analysis of licensed drivers in the USA, for reference) where they use both SQL and Python with visuals to walk through a case study.
Here is another example of Exploratory Data Analysis done in Deepnote by another analyst, Julie Whitton - Fruits and Veggies Data Analysis. It’s done using a free plan with limited features to give you a good look into what a data science notebook is.
Congratulations to Hyperquery and Deepnote on their new partnership. Let’s hope the team will revolutionize analytical workflow to a new, better standard for data science.
Entering Notebooks
I differentiate between two types of analysts: those who prefer notebooks and those who are more comfortable with sheets.
We are all trained (hopefully) to work with both types of tools, depending on the role, but it’s a challenge to do this cross-adoption.
It’s a challenge to convince the old-school BI, which has its roots in financial analytics, to adopt the So what? logical framework facilitated by notebooks. Conversely, it’s a challenge for data scientists who naturally prefer notebooks for deep dives, data exploration, and data storytelling to respect data precision and metrics checks, which are easier to do and maintain in sheets.
Bringing sheets and tables into notebooks would be groundbreaking in connecting these two worlds. However, Hyperquery and Hex's attempts to merge sheets and notebooks fall short for me. It’s like replicating Excel—you either have to carry all of its formulas, pivots, and conditional formatting or don’t bring them at all. With spreadsheets, I don’t believe there is a compromise of bringing 30% of Excel's features.
Why notebooks are the best for analytics
They offer version control, ensuring that analysts can track and manage changes to their analyses over time.
They combine R, Python, SQL, and Excel in a single IDE.
They are easy to distribute, making it simple to share insights with stakeholders or embed them into other documents, channels, or platforms.
They allow multiple analysts or teams to collaborate by reviewing each other's work, commenting, correcting, and supporting each other.
Notebooks let you store old analyses and documentation, making it easy to navigate to, access, or reference past work.
If you think about it, 80% of an analyst’s work is exploratory analysis, case studies, and root cause analyses. We work with data storytelling, where we can write an introduction, offer context, point to caveats, and format titles and sections. The main output of our work naturally fits into a notebook where we can share numbers, text, visuals, and commentary and back up every data point with a reference.
To make it work, we typically utilize dashboards, spreadsheets, and presentation decks as the primary output of our work, supplementing it with GitHub, where we store code, and WIKI/Notion, where we keep business logic and definition, and JIRA/Asana, where we keep project scope and description 😵💫.
Notebooks accelerate data storytelling, offering documentation, analysis, version control, code, reporting, distribution, and consolidated information all in one place. They indeed make it easy to follow the steps, review the code and findings, and collaborate.
Thanks for reading. Until Wednesday!
Don’t forget Observable Notebooks! They’re JavaScript-based, but in every way made for data. You can write SQL like in Deeonote (which I also love), create quick exploratory charts in Observable Plot or any other JavaScript charting library, and craft bespoke charts in d3.
No mention of VScode's notebook ?💀