Subscribe now

Today’s publication is about one of the most stressful meetings for analysts: the Weekly Business Review (WBR). Every company does it, since it’s the most standard way to check in on what’s going well, where attention is needed, and what we should escalate. Historically, it’s been driven by analysts or the head of reporting, because we’re typically the ones presenting performance metrics.

If you’re reading this newsletter, you’re likely very familiar with WBRs. What you might not realize is how many ways there are to run them, and how differently companies structure the cadence:

• An hour-long deep dive into every aspect of the business.

• A quick 15-minute metrics check-in.

• An async review in Slack, Teams, Basecamp (or email).

• A fully automated, templated summary delivered to teams in the morning.

Each approach has trade-offs, and I keep seeing companies combine formats - an in-person meeting plus an automated snapshot, for example.

Since I’m driving WBRs for most of my apps, I’ve been researching the best formats and best practices to keep them quick and productive, and I was surprised by how underrated this topic is. So I invited my friend Nick End (ex-Amazon, founder of RowZero) to share how best-in-class teams run WBRs: what tools they use, what cadence is best, what to pay attention to, and how to make these weekly check-ins productive and actionable.

Nick End is the co-founder and COO at Row Zero. He sold his first company, Shoefitr, to Amazon in 2015, then went on to lead product and engineering teams at AWS’s Confidential Projects org and at Tasso, a healthtech and medical device company.

I met Nick at SaaStr in San Mateo 3 years ago. He mentioned he was building a modern cloud data spreadsheet, and I was instantly skeptical and curious. I was also searching for a good spreadsheet tool. We kept in touch, and when they launched Row Zero 2 years ago, I remember trying it that very night and being impressed by how quickly it worked on large datasets. It’s in my toolbox now. I was just using it the other week for my ChartMogul data analysis of the impact of the number of plans on growth.

Highly recommend giving it a try - it’s intuitive, fast, and flexible. You can connect directly to your warehouse or simply upload a CSV. It saves your work and definitions, supports the same Excel formulas and pivots, processes billions of rows in seconds, and includes enterprise-grade security features to keep sensitive data off employee laptops. The team is growing fast too - last year they raised 10M in seed funding, launched an AI spreadsheet assistant, and have more features coming soon.

I’m excited and grateful Nick agreed to write for us (especially after just welcoming a baby 2 weeks ago!) and share what he learned at Amazon about how to structure, run, and operationalize successful Weekly Business Reviews.

Keys to a Successful Weekly Business Review

Many businesses operate some form of a weekly business review or “WBR” as it is commonly known. The WBR was made popular by Amazon as part of Jeff Bezos’s strategy to ensure the company is always tracking its plan for key metrics and KPIs. When done well, WBRs become the operating rhythm of your company - creating visibility, accountability, and alignment every week.

Much of the current Row Zero team worked at Amazon and has deep experience with WBRs. In fact, the Amazon WBR inspired the founding team to build Row Zero as a connected spreadsheet for Amazon-sized datasets. Below, I share details of our experience with Amazon WBRs and provide guidance for how to run an effective WBR.

What is a Weekly Business Review or WBR?

The WBR is a weekly structured meeting in which a team or organization reviews performance metrics, looks at trends, and makes decisions about urgent topics or weekly plans. Often, the metrics or KPIs are coded red, yellow, and green to indicate whether they are missing, trending in the wrong direction, or exceeding their goals. The objective of the WBR is to provide visibility into and accountability for the key metrics that drive the business on a weekly frequency to ensure it meets its long term objectives. The benefit of a weekly business review instead of a monthly or quarterly review is that it ensures the business can respond to deviating metrics before they become larger problems.

Why is Amazon known for the WBR?

While weekly business reviews pre-dated Amazon and were popularized by other companies, like GE and Toyota, the WBR became legendary at Amazon due to the company’s exponential growth and for the unique characteristics of Amazon WBRs:

1. A data-driven narrative focus - During Amazon WBRs, teams review 10s or 100s of metrics each week to identify problematic trends before they grow into larger issues. Documents are prepared before the meeting for everyone to read and question. No PowerPoint slides or generic high-level statements.

2. Personal accountability - Every metric or KPI has an owner responsible for it. Owners come to the meeting prepared to explain what happened and why.

3. A ritual - The WBR became a core cultural ritual that has helped Amazon grow exceptionally fast and operate at a high level across massive teams and product categories.

How does Amazon run its WBRs?

Amazon typically holds its org-level WBRs (e.g., retail, AWS, etc.) on Wednesday, allowing reporting teams under subordinate VPs or directors to hold smaller WBRs on Monday and Tuesday. This structure allows ample time for subordinate teams to roll up their WBRs and be prepared for the org-wide WBR each Wednesday. As an example, the storage team in AWS S3 holds its WBR on Monday, the S3-wide WBR is held on Tuesday, and the AWS-wide WBR is on Wednesday.

Here is the typical agenda for WBR at Amazon:

• The WBRs start with 10 minutes of document reading. I won’t go into much detail on Amazon’s document culture because it has been well documented (pun intended). In short, Amazon meetings always start with a 1-6 page document (with unlimited pages of appendices, often 10s or sometimes 100+ pages).

• Everyone in the meeting reads the document and makes notes in the first 10 minutes.

• Then the questions begin. In the case of WBRs, the document owner writes a narrative calling out metric fluctuations and justification for deviations from the plan.

• Subordinate teams often rotate the owner of their WBR so a different person gets the experience (burden) of compiling the WBR each week.

• Oftentimes, a list of specific metrics and their trends are included in the printed document, but are also often reviewed in a data tool.

When I worked at Amazon, it was fairly common to use spreadsheets to track the long list of metrics. Anecdotally, one of the Amazon retail WBRs and the Amazon Alexa WBR were run off a massive Excel spreadsheet. The Alexa WBR Excel workbook took a half day to update due to the dataset size. The advantage of a spreadsheet is that anyone can use it, understand it, and make their own copy to drill down and investigate. Other groups use BI dashboards or custom-built analytics portals. A line of questioning will often ask for additional detail, in which case it is nice to quickly drill down to pull out relevant insights by customer, product, sales channel, or other criteria.

In the AWS org-wide WBR, the group is so big that it isn’t possible to dive into each group every week. Instead, AWS spins a wheel (or at least they did when I worked there) to pick the department that gets the spotlight from AWS leadership that week. Sometimes the wheel lands on your team 2 weeks in a row! Every group needs to be prepared to present, especially if your team is not hitting its goals.

Part of the power of Amazon WBRs is the detailed structure and accountability.

Metrics reviewed in WBRs were more granular than you’d expect. For example, in the Amazon retail WBR, sales, returns, operating profit, and other metrics were monitored for every brand and product line permutation. Vendor managers were responsible for providing data-driven explanations when metrics deviated from plan. For example, deviations might be due to promotions of competitive products, high returns, low inventory, or other factors, and any vendor manager needs to explain what happened in the previous week.

If someone asks a question and the metric owner doesn’t know the answer, the question is written down, and the owner is expected to follow up by email within a day.

In cases where metrics are continuing to deviate from plan, corrective action plans are often put in place to help remedy the situation.

This consistent visibility and accountability foster a data-driven and results-oriented culture that has powered Amazon’s success for decades.

How to implement and run a WBR

To implement your own WBR, you need 3 things:

1. The data to track key metrics or KPIs driving your business.

2. The tool(s) to track these metrics on a weekly basis.

3. The people responsible for those metrics.

WBRs can be for a small team, a business unit, an organization, or an entire company. Here are the steps to get started:

1. Decide on your metrics

Collaborate with business stakeholders to determine the metrics to track and with stakeholders to verify accurate source data for those metrics.

Ideally, you want to be able to track actual results vs goals vs prior week’s results. While consistency is important, be willing to add or remove KPIs to achieve your goals.

2. Set up your weekly metrics tracking

Once you’ve identified your metrics and source data, set up a connected spreadsheet or dashboard to track your metrics. For example, you can connect a Row Zero spreadsheet to your data warehouse - this keeps metrics automatically updated and also makes it easy for anyone to drill into the underlying data to investigate issues and opportunities. You can also use add-ins for Excel/Sheets if not working with large datasets. Avoid manual file uploads or copy/paste workflows to keep maintenance efficient, accurate, and secure.

Use conditional formatting to automatically format metrics to Green, Yellow, or Red to make it easy to see positive and negative metrics.

3. Review your spreadsheet/dashboard with stakeholders

Before hosting your first meeting, review your WBR spreadsheet with your business and data stakeholders to ensure you’re aligned on the right metrics and data sources. You don’t want these folks questioning the accuracy of the data in your first meeting - you want them questioning the reasons and implications of the metrics. Iterate if needed until you get to alignment.

Also, ensure every metric has an owner and ensure metric owners have direct access to the raw, underlying data, not just top-line numbers. This allows them to investigate root causes and come prepared with answers.

4. Schedule the Meeting

Put a recurring weekly meeting on everyone’s calendar. Most teams hold WBRs on Monday or Tuesday to review the prior week, but choose a day/time that makes sense for your org. Keep the meeting focused, but give ample time to review and discuss the metrics.

Plan for 30 minutes for smaller orgs, but larger teams will need 60 minutes (or longer).

5. Run the meeting

Typically, the data owner who compiles the metrics facilitates the discussion.

Review each metric methodically: Red, Yellow, or Green. Each metric owner should explain performance and any corrective actions, and attendees should feel free to ask questions. If metric owners don’t have an answer, they’re expected to follow up in writing after the meeting.

6. Track actions and follow-ups

Assign someone to keep a running log of follow-ups and action items. Send meeting notes afterward and check in on open items at the next WBR. Consistent accountability builds momentum and positions you as a leader for driving the process.

WBRs week to week

When done well, WBRs become core to your culture. They drive decision-making and keep teams aligned and accountable. If set up with a connected spreadsheet or dashboard, it takes no effort to update your core weekly metrics, and any prep time should go into reviewing metrics, investigating anomalies, and crafting a clear narrative for the week.

A lot of power and visibility come with owning the meeting and owning metrics, so WBRs can be a good opportunity to take a leadership role in your company. And don’t worry about getting it perfect right out of the gate. WBRs are an iterative process. The key is just to get started. Each week is a new chance to improve.

Thank you, Nick!

Find Nick:

• LinkedIn: https://www.linkedin.com/in/nick-end/

Learn more about RowZero:

• RowZero: https://rowzero.com/

• Info on connecting Row Zero to your data warehouse: Databricks, Snowflake, and others. RowZero documentation: https://rowzero.com/docs.

• YouTube: https://www.youtube.com/@Row-Zero

• If you are a student or recent graduate, you can get a free Pro subscription. The platform lets you explore large public datasets, collaborate with others, and publish your research

Bookmarked case studies from Row Zero Blog:

• AWS Rolls Out Row Zero to Expand Self-Serve Analytics

• Don’t send data to people. Send people to data.

• How to Calculate Percentiles in Pivot Tables

• Run ARIMA in a Spreadsheet

• How to automate spreadsheet updates

• How to build a waterfall chart

Subscribe now

If you’ve been reading my newsletter, you know I’m a fan of notebooks. I’ve used them since the early Jupyter days and tried almost every notebook out there. It’s been interesting to watch how they keep reshaping data science and what it means to be an analyst.

For a long time, Deepnote was my go-to (Data Portfolio Done In Notebook). A few months ago, I switched to Hex, and it completely changed my perspective on what notebooks can and should do. Hex is setting the bar very high.

Before diving into my own analyses and projects in Hex, I wanted to start with a conversation with its founder about what Hex is, why it was created, how its vision has evolved, and what skills the next generation of analysts should build.

Hex is one of the most well-known applications in analytics today. It’s a notebook that seamlessly combines SQL, Python, and interactive visualizations, allowing data scientists and analysts to explore data more effectively. They don’t call it a “notebook,” though - it’s a “connected platform for using AI to work with data.” Whatever you call it, more than 1,500 teams worldwide use Hex, and a few months ago, the company announced $70M Series C round to fuel the era of agentic analytics.

A few weeks ago, I stopped by Hex’s office in San Francisco - the one with a framed BlackBerry 8130 (because apparently nostalgia is part of their brand?). At first, I was a little intimidated - if you Google the Hex founders, it’s giving “classified government project, but make it SaaS” vibes:

Turns out, they’re actually fun. Though, to be fair, I can only vouch for Barry.

So, everyone - meet Barry McCardel, co-founder and CEO of Hex.

How did it all begin? What is the Hex story?

I’ve been a builder and lifelong user of data tools. My co-founders and I all met at Palantir. Every job I had, the pattern was the same: the stack existed - warehouses, BI tools, but 80-90% of our time went into maintaining dashboards, while real analysis lived outside them.

Then, at a healthcare startup, the first question leadership asked was, “How’s enrollment?” - something a dashboard could answer. The next question, “What happens if we change X?”, required chaos: running queries, exporting CSVs, opening Jupyter, generating charts, taking screenshots, and pasting them into decks. Totally inefficient.

I started out looking to buy something better. I couldn’t find anything - none of the open-source tools or commercial products really solved it. My friend said, “Maybe you should build it.” I laughed, then realized he was right. I hadn’t planned to found a company, but we, my co-founders Caitlin and Glen, had the right experience and taste to build the kind of tool we always wanted, so Hex was born.

We raised seed funding in 2020, once we had early users and a working product. The biggest early challenge for us was focus. Data is infinite. The hardest part is deciding what not to build. It’s very easy to spend years building something no one will use. Customers don’t always know what they want - follow every request and you get a Frankenstein product, but ignore them and you lose relevance. Balance is key. I wrote about this balance in a post called Commitment Engineering. It’s about making deliberate product tradeoffs.

We built Hex selfishly, the tool we always wanted. I still use it daily. Our 150-person team does too.

Why a notebook?

In our early VC pitch decks, I wrote: “There’s no standalone market for notebooks.” The notebook is a format, not a product. The goal is an integrated environment - analysis, visualization, and collaboration in one workflow.

Other industries solved this - Figma integrated design workflows, Notion unified docs and databases. Data deserves the same: one environment instead of a separate BI tool, notebook, and AI assistant.

We even printed notebooks embossed with “Not Just a Notebook.” Ironically, by rejecting that label, we built the best one.

There isn’t a single “notebook market.” VS Code’s notebook serves engineers. Jupyter serves researchers. Hex serves analysts and data scientists - people moving between SQL, Python, and visualization.

Like dbt defined “analytics engineering,” we’re defining a space for integrated analysis - where exploration, modeling, and storytelling happen in one place.

Most Hex users never touched Jupyter. The barrier was too high: local setup, package management, broken environments. We didn’t commercialize Jupyter like others tried, we solved a broader problem: letting anyone who asks questions of data work productively. We made notebooks usable for analysts. SQL cells, chart cells, and reactive DAGs were our core innovations. We turned a developer tool into an analyst’s environment.

Any comment on the Hashboard acquisition? That was quite a surprise.

I’ve known Carlos (Hashboard’s founder) for years. Hashboard focused on BI, made thoughtful architectural choices, and punched above their weight with a tiny team (like 8). We began talking about working together about a year ago. Integration has gone exceptionally well. Their team is contributing to recent releases, and Carlos now has a larger product role here at Hex.

I met Carlos and the Hashboard team, and agree - they’re great. After seeing the demo, I was impressed by how well they handled version control and how straightforward they made dashboard maintenance. Really well done.

How does AI fit in notebooks?

We launched our Notebook Agent a few weeks ago, and the adoption has been explosive. The notebook itself is an amazing format for iterative, deep analysis – and agents work into that pattern really well.

You can ask a question, and watch as an agent builds you the first draft of an answer using all the cells and context in Hex – it’s really amazing.

Once you start working this way you just can’t imagine going back. And now there’s a whole new frontier of techniques and methods for working with the agent in Hex – it’s so fun.

We also just launched Threads - a conversational UI interface built on the same framework as our notebook agent, but simplified for non-technical users. It’s the world’s best way to do self-serve. It’s more “on the rails”, without access to Python, restricted to endorsed data, and a preference for semantic models. For charts, it uses Explore cells, so users can drill in. If you send a thread to the data team, they see the notebook behind it. This bridges non-technical users and analysts.

And of course, you want to trust all the answers you get – so we have a semantic modeling agent. It isn’t a GPT wrapper, it’s built into workflows. You can point it at a Hex project to bootstrap a model, ask “what should I build next,” and it will propose and draft additions based on your warehouse and projects. You review, fix minor issues, and publish.

This forms a virtuous cycle:

• Editor (data team): notebooks for new/gnarly work.

• Curator: publish models and apps as a curated context.

• Explorer: self-serve answers draw first from existing models/projects; escalate back to editors when needed.

Our product org mirrors this: Editor team, Curator team, Explorer team, plus a platform team (agents, infra). We also dissolved a separate “AI team.” Now every team is an AI team.

Is Hex evolving compared to the team’s initial vision?

BI itself is evolving. AI hasn’t really changed data work yet. There are lots of demos, but little durable workflow. With our Notebook Agent, we aimed for workflows that actually work, not a thin GPT wrapper.

Our approach remains the same - build workflows that actually work, not gimmicks. That’s our north star: make analysis powerful and accessible.

So evolution is to bridge notebooks, BI, and AI in one collaborative workflow.

Exactly. We want to make data analysis both powerful and accessible, for everyone from SQL analysts to decision-makers.

We get requests for classic BI features (new charts, drilldowns). We’re building some, but the wrapper, how answers are created, audited, and consumed, will change. Hashboard accelerates this through their expertise, not by grafting their product on top.

Do data scientists need to learn SQL and Python today?

Memorizing syntax matters less. Understanding systems, modeling, and debugging matters more.

AI can write code, but not reason about data. Analysts should learn by building projects over courses. Employers care more about thinking, prioritization, and communication than knowing every function name.

Syntax memorization matters less. Conceptual understanding matters more: systems, data modeling, how pieces fit, how to debug when AI-written code breaks. Learn by building projects and using tools, not by rote. Employers value how you think, prioritize, spot problems, and communicate with stakeholders over trivia about language syntax.

Thank you, Barry!

Find Barry:

• LinkedIn

• X

• Barry’s blog

Learn more about Hex:

• Hex - https://hex.tech/

• Hex product guide - What is Hex?

• Hex blog - https://hex.tech/blog/

Subscribe now

A few weeks ago, the analytics world got big news: Mixpanel appointed Jen Taylor as its new CEO, succeeding Amir Movafaghi.

For anyone working with user interaction data, Mixpanel needs no introduction. It’s one of the most established analytics tools, used by more than 29,000 companies worldwide, and has had a huge impact on product and marketing analytics.

I’ve been using Mixpanel since 2012, and it’s shaped the way I work as an analyst. I still use it a lot today, following its features, updates, and announcements closely. So when I heard Jen was stepping in as CEO, I wanted to understand what this means for Mixpanel, its users, its competitors, and the analytics space overall.

Two weeks ago, I had the chance to meet Jen for a candid conversation about the state of product and marketing analytics, how AI is reshaping the field, and how Mixpanel is positioning itself to embrace this change and drive growth.

Jen Taylor served as President at Plaid and Chief Product Officer at Cloudflare, where she helped grow the company from $100 million to $1 billion. Before that, she led product and data initiatives at Salesforce, Facebook, and Adobe. Now she is joining Mixpanel, and I’m (selfishly) hoping we’ll see more new features, more success stories, and innovation that accelerates analytics without compromising trust or accuracy.

Today, I’m excited to share my interview with Jen, along with her perspective, values, and vision for how Mixpanel will lead and transform analytics.

What are the next steps for you and for Mixpanel?

Part of what drew me to Mixpanel is my long-standing passion for understanding customers and users. I’ve spent most of my career as a product manager, and my true north has always been the customer. Tools like Mixpanel have been transformational for me because they allow you to marry quantitative behavior with qualitative insights in a way that gives a much deeper understanding of the user.

Right now, I see 2 forces that are especially transformational:

1. The market is shifting away from looking at product analytics in a silo.

With advances in data warehousing and infrastructure, we can now combine product analytics with other data sources across the business. That means getting a more complete picture of the customer and driving value not just for product teams, but also for business and financial stakeholders. Mixpanel has a huge opportunity to accelerate that shift.

2. We’re in a moment where AI can fundamentally change how analytics work.

For Mixpanel, AI can make implementation easier, strengthen data governance, and speed up how people turn questions into dashboards. More importantly, it can shift analytics from reactive, where users pull insights, to proactive, where the system surfaces insights, suggestions, and areas to focus on.

For me, the exciting part is listening to customers and working with the team to figure out where those changes create the biggest opportunities or challenges, and how we can refine our strategy to meet those needs.

Amplitude just launched AI Agents. Is Mixpanel planning to launch similar AI features?

Do customers even ask for it?

Mixpanel has been working with machine learning and AI for some time now. Over a year ago, we launched Spark AI (Ask Spark AI), and more recently, we’ve released a set of new technologies that are still in beta as we continue exploring how to integrate AI throughout the platform.

I see our role in AI in 2 main ways - applying AI to the real problems we hear from our customers and see in the market, and then it’s about helping our customers learn, adopt, and get value from AI.

AI is incredibly powerful, but it takes time to understand and even more time to build trust in the results. Part of our journey at Mixpanel is to meet customers where they are today and support them as they grow with these technologies.

What is your long-term vision for Mixpanel? How are you thinking of differentiating from other digital analytics tools?

I absolutely see us continuing to refine and define who we are, and that will be based on how we understand the opportunities and needs of our customers, and how we position ourselves within the broader ecosystem of tools and solutions they use.

Most of our customers work in heterogeneous environments - using Mixpanel alongside other data platforms, financial systems, and more. The question for us is: how do we fit within that ecosystem, and how can we serve as a multiplier for our customers and their products?

One of the things that drew me to Mixpanel is its continuous focus, since day one, on democratizing action and insight. As a product manager, I’ve seen how much organizations aspire to that, but also how hard it is to achieve. Given the technical moment we’re in now, I believe Mixpanel is uniquely positioned to lean into it as a real differentiator.

The team has been cranking out innovation, expanding into session replay, experimentation, and more. Just before I joined, we launched metric trees. We’re constantly working with our users to understand their needs and how we can broaden the capabilities that help them better understand how people are using their products.

Mixpanel has many new great features! Even as a power user, I haven’t had the chance to try some.

That’s a classic software conundrum: people get to know a product at a certain point in time, and then the challenge becomes - how do you help them discover and use new capabilities?

They tend to stick with what they already know. Our job is to keep evolving and improving the features they use today, while also introducing new ones that could be incredibly valuable to them.

Are you thinking about expanding the team, maybe even setting up more headquarters in different regions for Mixpanel?

Right now, I’m in listen-and-learn mode. I’m getting to know the team and our customers. Our strategy around locations and the markets we serve will be completely driven by where we see demand and where we find talent.

One of the things I really enjoy about Mixpanel is that it’s a truly global organization, with offices in San Francisco, London, Barcelona, Bangalore, and Singapore. Our customers are all around the world. That global reach is both important and impactful. I also believe that the diversity of experience is critical.

What values do you want to bring to your team at Mixpanel?

A big part of what helped me through the decision-making process to join Mixpanel was feeling that the culture and values of the company were very aligned with who I am and how I lead. That gave me confidence that I could come in as a new leader, listen and learn, and work within a culture that’s already incredibly successful, while also thinking about how we can continue to evolve and grow together.

The collaboration, teamwork, integrity, and empathy I see within the organization are all closely aligned with how I lead. And, of course, there’s the strong focus on customer centricity.

Thank you, Jen!

Closing remark:

Certainly, I’m impressed with Jen’s experience and her openness to feedback. We also discussed some of Mixpanel’s features and analytic challenges, such as maintaining and enforcing taxonomies or metric trees, and I can tell Jen feels the product and knows the space very well.

The biggest value analytics tools can offer is recognizing your product’s unique structure and value proposition, and then tailoring the analytics experience to fit your specific footprint. The challenge is how to make general frameworks work for a particular product. How to adapt copy-paste templates to your own data. That’s hard to do.

This is where AI can make a real difference - by recognizing your event structure, data volume, organization structure, and product type, then guiding you through analytics implementation and consumption, helping you take full advantage of the features and frameworks available today.

Would love to see Mixpanel manifesting it.

Learn more about Mixpanel:

• What is Mixpanel?

• The Guide to Product Analytics

• Learn to Use Mixpanel

Learn about event data analytics:

• Introduction To Event-Based Analytics

• How To Set Up Analytics for Web and Mobile Products

• Inside Product Analytics: Decoding User Behavior

• Inside Product Analytics: Decoding User Behavior Part 2

Subscribe now

If I could choose any new analytics path for myself, it would be data storytelling, without a doubt. Data storytelling combines data analysis with design and communication. Dry numbers become clear and compelling stories, both visual and non-visual.

I’m especially drawn to data storytelling because it’s a skill I’ve always lacked. As a typical data person, I think in structures and tables, so my writing often ends up dry, full of lists and bullet points, and hard to read. That’s just how my brain works. Honestly, I’ve never been able to write seamless stories like , Casey Newton, or - or like one of the best data storytellers out there, Chris Dalla Riva, whom I recently had the chance to interview.

Chris is a musician, data analyst, and writer. He lives at the intersection of music and data, focusing on analytics and personalization. He writes the popular newsletter Can’t Get Much Higher and has been featured in The Economist, Business Insider, and NPR, among others. He also just published a book, Uncharted Territory: What Numbers Tell Us about the Biggest Hit Songs and Ourselves, where he presents a data-driven history of popular music in a way that’s both fascinating and accessible.

Below, I’m sharing my interview with Chris on how to become a data storyteller: how to develop an “eye” for stories, what skills and tools to focus on, and how the field is evolving. You’ll notice how easy his writing is to follow and how naturally he draws you in. I’m absolutely fascinated by how he communicates in such an engaging and welcoming way.

Can you share your story of how you got into data journalism and what fascinates you about it?

Looking back, it makes sense that I would try to do something with data journalism. I studied mathematics and economics in college. I always had an interest in writing. I loved learning about music and culture in my free time. But actually getting into data journalism was kind of an accident.

Years ago, I pitched a story to a website called Tedium about how there were fewer key changes in popular songs.I thought it was quite niche, but it ended up going mega-viral, leading to me appearing on NPR and in The Economist. That made me realize that people might be interested in what I had to say. So, I decided to start up a newsletter where I could write about music and data every week.

I feel like that key change story is a good example of why I like data journalism. I always felt like there were fewer key changes in songs these days, but data makes it possible to check. Often what frustrates me about certain stories – in music or not – is that claims are made that could easily be verified. Of course, there are limits to what data can do as some things are not easily measurable, but I think it’s nice to try to measure things when we can. It at least gives us a firm starting point from which we can agree or disagree.

How do you define data journalism, and what are some typical examples of it?

For me, data journalism is any form of reporting or analysis that includes something quantifiable. Of course, that includes people using sophisticated statistical methods, like you might see Nate Silver or other political pollsters do, but it also includes much simpler things, like surveying 100 people about some topic and reporting on the results.

Sherwood News has a lot of great examples of the former. For example, here is a short piece they did on drinking habits in the United States. If you know what a percentage is, you can understand this piece. Still, it’s very insightful. I think people assume that to be worthy of the moniker “data journalism” a piece must be complicated. I don’t think that’s the case. In fact, I think data is supposed to make things clearer if used correctly. Most of my pieces are just using sample means and ratios.

When I think of great data journalism, here are a few pieces that come to mind:

• The Largest Vocabulary In Hip Hop

• Why Classic Rock Isn’t What It Used To Be

• Unraveling Florida Man: The Meme, The Myth, The Legend. A Statistical Analysis.

• Elon Musk’s Hard Turn to Politics, in 300,000 of His Own Words

• The Strava Heat Map and the End of Secrets

• John Snow's data journalism: the cholera map that changed the world

• The Save Ruined Relief Pitching. The Goose Egg Can Fix It.

• A history of the American economy through stadium names

What are the biggest challenges you face in your work?

My biggest problem is getting data. I know many data journalists who find existing datasets and build their reporting about what they can mine from those datasets. Of course, I’ve done this too, but I typically start with a question and have to figure out how to get the data to answer it. Sometimes this results in me scraping Wikipedia and pairing that scraped information with data pulled from Spotify. Other times, like in my book, it requires monitoring hundreds of terrestrial radio stations.

Sometimes there just isn’t data available, though. For example, a friend of mine has this theory that children of divorce are more likely to pursue careers in music, with a particular concentration in punk music. I think it’s a great theory. And my gut says it’s true. But it’s really hard to find divorce data that can be paired up with popular musicians. Sometimes the biggest challenge is knowing when you shouldn’t weigh in on a topic because the data doesn’t exist.

How do you choose topics or themes for your analyses?

If I am not truly interested in a topic, I usually can’t bring myself to write about it. (The nice thing about writing for yourself is that you don’t have to write about things that don’t interest you!) But the way that I find things that interest me come from many places.

Sometimes I get a topic from a friend or coworker, like when my coworker asked me who the greatest two-hit wonder of all-time was. Other times, I’ll give my take on a prominent news story, like when an AI band got millions of streams on Spotify. That said, most of my topics come from a little notebook that I keep on my desk. Whenever something vaguely interesting strikes me, I put it in there. Then every Sunday or Monday, I pick one to write about. Some that are in there now are the following:

• Which US state has the best musical Mt. Rushmore?

• How have the songs played at funerals changed over the decades?

• Did Guitar Hero impact the musical tastes of a generation of people?

• Have the colors we see on album artwork changed in the last 20 years?

Feel free to steal one of these ideas if you’d like. I’ve got too many things going on and will probably never get to them.

What has been your favorite piece of coverage or analysis so far?

If we are talking about my favorite pieces that I’ve done, the number one spot has nothing to do with analytics. Basically, my grandmother would always claim that Frank Sinatra performed at her high school while she was a student there. She was a straight shooter, so nobody had any reason to doubt her, but it was quite the claim. After she died, I wanted to see if there was any evidence that the show happened. So, I went on a long quest looking for proof. I spoke with Sinatra experts, dug through newspaper archives, and bothered more than a few strangers on Facebook. I won’t spoil the ending, but it was quite a journey.

Anytime I meet someone that has read my work, this is always the piece they bring up. I’ve always found that interesting because it’s so different from my normal data-driven pieces. At the same time, I think my training in data analytics made this investigative piece easier. Data journalism is all about sleuthing through sources, albeit those that are often numeric. But those sleuthing skills are applicable in many other domains.

What is your upcoming book about?

In 2018, I decided that I was going to listen to every number one hit song in history. Part of this decision was driven by my interest in music. I loved playing in bands and writing songs. But another part was driven by desperation. I was working a stressful job that I didn’t like. 3-minutes with a song at the end of each day was a reprieve from my day-to-day life.

Given that I’ve always worked with data, I started collecting tons of information about each song. Which label released it? What was the gender of the songwriter? Was the song entered in the Eurovision contest? From this ever-expanding dataset – which is now publicly available here – I started noticing some trends. Inspired, I wrote some stuff up about what I was seeing.

These stray observations slowly evolved into a data-driven history of popular music covering the period from 1958 to 2025. It’s called Uncharted Territory: What Numbers Tell Us about the Biggest Hit Songs and Ourselves. It’s packed with great visualizations made by my friend Caileigh Nerney, like the one you can see below.

Can you walk us through your process for creating data-driven analyses? What tools or software do you use, and how long does it typically take?

I’ll give you an example from a piece that I wrote. Everything starts with a good question. In this case, my question was simple: what is the most underrated song of the 1980s? From a question, you need to figure out how to measure what you are looking for. In this case, I decided that a song is underrated if there is a big difference between critical and popular perceptions, meaning critics loved a song much more than your average person.

With a methodology, you then need data. In this case, I used Spotify stream data to proxy popularity and average song rankings from 7 critical lists to proxy greatness. I then compared them to get the results I was looking for. As in most cases, I manipulated the data with Microsoft Excel and Python’s Pandas analytics package. I then visualized with Data Wrapper.

End to end, this piece took me a couple of hours across two days. My typical timeline is that I start writing on Monday and publish on Thursday each week. Occasionally, I start a piece earlier if I need more time, but I generally work fast and stick to a publication schedule.

How do you see data journalism evolving, and what might it look like in the future?

In the last 15 years, data has come to touch basically every aspect of journalism in some way. As things continue to move online, I think our world will become more quantifiable. I expect data analysis will become an even more fundamental skill for many parts of the writing process. I also expect this writing to be more interactive, like the style pioneered by The Pudding. If you’re unfamiliar, I did a piece with them about quantifying the greatest album of all-time. I also really loved this piece they did about the evolution of the literary canon.

What skills or qualifications do analysts need to succeed in data journalism?

While I always tell people that it’s good to have a working knowledge of statistics and the ability to write some code, especially in Python and SQL, I think the most important skill in being an effective data journalist is the ability to perform rote tasks without losing your mind.

That probably sounds crazy. But even in our highly automated world, so much of data analysis is data cleaning. You’ve got to be willing to get down and dirty with your data if you want to find interesting insights. Sometimes that requires writing a few lines of SQL. But other times it requires building and cleaning lists of thousands of television theme songs from Wikipedia, like I did in a piece a few years ago. It’s not as sexy as running regressions, but I think it’s where the deepest insights lurk.

Is there anything else you’d like to share to inspire people to learn data or pursue a career in this field?

If you have a question that you want answered – with data or not – don’t be afraid to start searching and publish what you find online. 99% of opportunities I’ve had as a data journalist have come because I sent a cold email to someone who I never thought would reply or because I started publishing my thoughts before I was technically ready to. Don’t be afraid to shout into the void. You never know who is listening.

Thank you, Chris!

Find Chris:

• LinkedIn

• X

• TikTok

• Substack

• How to Report on DAU/MAU Ratio - A refresher on the most common engagement measurement - DAU/MAU ratio: how to interpret it, DAU/MAU benchmarks, examples, and how to align it with your product’s real usage frequency.

• Which Analyst Are You? - The generalist data analyst role is retiring, giving way to specialized analytics positions. My short career guide for analysts: an overview of roles, skills, and industry trends.

• Semantic Layers: The Right Idea, the Wrong House - A skeptical take on semantic layers (until you truly need one). Learn the difference between semantic and metrics layers, when and why to use each, and how to decide if your team is ready—or if you’re better off without one.

Subscribe now

This newsletter is a little special ⭐

Once per year, I reach out to my network of analysts and data experts with one question: What is the best advice you received that changed your career? Today, I want to share some of the wonderful responses I received. Some of these are personal stories, others are quick dos and don’ts. I hope they inspire all of us to grow in analytics and become better at what we do.

I wanted to kick it off with a few words from Nico Rosberg. Nico is a German former Formula 1 driver and entrepreneur who won the World Drivers’ Championship in 2016 with Mercedes, securing 23 Grand Prix wins over 11 seasons. This isn’t about data or analytics, but his recent LinkedIn post really made me think:

“Only 10% of my races in Formula 1 ended in victory.

That means I lost 90% of the time. And yet, I still became a World Champion. So what kind of mindset does it take to keep going?

For me, the answer was mental training. Learning how to reset, refocus, and turn every setback into a stepping stone. That was the key to reaching the top. […]

In F1, I mastered mental training, and one of the methods that I was taught by a mindset expert was to snap my fingers after a bad race to reset. That one snap meant: move on, refocus, and prepare for the next challenge.

Now in business, that same mindset is so helpful! The ability to recover quickly and stay consistent over time is way more powerful than chasing perfection.

It’s the same whether you’re reworking a product launch that missed the mark or regrouping after a no from a key investor. What matters most is how fast you reset, learn, and show up stronger.”

What Is the Best Advice You Have Ever Received?

Maxime Beauchemin - CEO & Founder at Preset; ex-Lyft, Airbnb, Facebook:

“Work in the open" - publish blog posts, get involved in open source, share discoveries broadly. Don't confine your impact to the boundaries of the company you work at!”

Barry McCardel - Co-Founder & CEO at Hex:

“Your work only matters if it changes decisions. If you’ve done the perfect analysis or beautiful dashboard but no one is doing anything differently because of it, you might as well just played hooky. Every analyst should know the exact decisions they want to change, how they are made, and how to measure the outcome. If you don’t have that you need to stop everything you’re doing until you have it (or find a new job!)”

June Dershewitz - Data leader @Amazon, co-founder @InvestInData, ex-Director of Analytics at Twitch, and the author of Measurecraft newsletter:

“Don’t just take the job that seems right today. Take the job that will set you up for where you want to go next.”

A mentor shared this with me earlier in my career, at a time when I was fortunate enough to be considering three different job offers for data leadership roles.

Building on a foundation of versatile technical skills, there’s a wide range of paths that data folks can follow. B2B or B2C? Marketing analytics or product analytics? Infrastructure or business-facing? The choices can seem overwhelming. That simple piece of advice helped me find confidence in the moment as I made my selection. It also helped me build discipline around long-term thinking, which is something I continue to carry with me in my career.

Here’s some bonus advice, although it wasn’t something I asked for. Fresh out of academia, I moved to San Francisco during the first dot-com boom. I wanted to be a software engineer. Instead, a startup offered me a job as a data analyst. I took it. It turned out to be an excellent fit. They saw something in me that I hadn’t yet seen in myself. That decision, back in 1999, set the course for my entire career in data.”

Randy Au - Quantitative UX Researcher @Google, the author of Counting Stuff newsletter:

“I don't think a specific single person spelled this advice to me, but I kept hearing and seeing variations on this theme until I've started telling people myself.

As data folk, we mostly learn how to arrive at rigorous, or at the very least, reasoned arguments and analysis to make our points. There's a thousand different ways to be right to varying degrees, and an infinite ways to be wrong. We naturally spend lots of time planning, thinking, and checking our work to try to be as correct as possible. But in industry, the people we are advising are often paid to make decisions under uncertainty. They need to make a decision and will do it with or without us. A "correct" analysis that misses the deadline will have zero influence on the decision. So one of the most important things to learn on the job is figuring out when to trade rigor for speed, and when to insist that rigor cannot be sacrificed for speed.”

Francois Ajenstat - Chief Product Officer at Amplitude, ex-Chief Product Officer at Tableau:

“Don't give a f*ck

The biggest limiter of success is often yourself. It’s easy to give in to the fear of failure, the fear of looking bad, or the fear of saying something stupid. My advice: care less about how you're perceived. Instead, care deeply about the outcome.

Worrying about what others think of you will only hold you back from reaching your full potential. Sometimes I say that as long as I have conviction in my work, I don't give a f*ck what people think

Adopt a beginner's mind

Adopting a beginner’s mindset is a core principle I learned from Marc Benioff during my time at Salesforce. Often, you need to take a step back and look at everything like it’s your first day on the job. Don't worry about how your colleagues will react. Ask hard (or embarrassing) questions, challenge tried and true playbooks, and overturn assumptions. Evaluating work with a beginner's mind is an important part of continuing to learn and grow. “

Reflecting on some advice from previous years:

Joe Reis - Data engineer and architect, author, global keynote speaker, professor, advisor, investor, podcaster, and writer of Joe Reis newsletter:

“Careers don’t often make much sense. Mine certainly doesn’t. You’ll probably look back on your career and wonder how you got to where you are today. And you’ll be confused about where to go next. This is part of the journey.

Here’s some advice that helped me navigate the challenges and uncertainty in my career.

• Reputation matters. People have long memories, and the world is a small place.

• Success isn’t so much about being smarter than everyone else; it is about avoiding stupid things that will put your career in the graveyard. Lying, stealing, cheating, and nastiness are a few ways to kill your career. If you look at the graveyard of careers, you’ll find many supposedly smart people buried there. Don’t be one of them.

• Continuously learn. Read, talk to people, write, and explore new ideas and skills. Learning compounds over time. The person who continuously learns is a force of nature.

• My boss once told me, “When I ask for the time, don’t tell me how to make a watch. Just tell me the time.” He meant that I needed to give a direct and clear answer when he asked a question. Clear and concise communication is key.

• Lift others up. The more people you can help, the more they will help you when needed.”

Carlos Aguilar - Founder/CEO at Hashboard, ex-VP Data @Flatiron Health, Advisor, ex-Amazon Robotics

“I worked in supply chain for about five years and the best advice I received from an early manager at the time was to zoom way in and dig deep when troubleshooting a problem. My natural tendency at the time was to look at the data as a whole and miss the human element.

If there was a warehouse that was operating poorly, just go look and see what's going on at the warehouse. Go talk to people, inspect the shelves, look at the actual products. The best data people are able to operate at multiple levels: zooming way out and thinking about the data as a whole and also zooming way in and not losing empathy for what is happening on the ground.

The greatest thing about data is that it can scale empathy, so don't lose sight of the fact that each event could represent a person (or a product, etc.)”

Ryan Janssen - CEO/Co-Founder at Zenlytic, entrepreneur, data scientist, and investor:

“Have intentionality.

Early in my career, I asked my mentor to help me choose between two very different jobs. I had a big pro-con matrix and a financial spreadsheet. I talked his ear off for 10 minutes before he interrupted and asked "But how do you want to spend this next 5-year cycle of your career?" And I realized I couldn't answer him clearly.

Many people think they have a good idea what they want to get out of their career, but actually don't. Or they continue on a path that clearly lacks the trajectory they need to get there.

My advice is to take time to introspect. Schedule time if you have to. Write things down. Be deliberate and detailed about what you want to do in your career. Be deliberate about the most important projects you need to work on now to get there. Careers are short and it's easy to let time slip away; for satisfaction and success, have intentionality about your career every single day.”

Kyle Poyar - Operating Partner at OpenView, the author of the Kyle Poyar’s Growth Unhinged newsletter:

“Look for ways to replace yourself. Early in your career, you get ahead by being the best analyst -- the person who knows the data better than anyone else. That quickly changes. To truly advance, you need to find ways to hand off your responsibilities to someone else.

It'll be painful in the short term. You'll feel uncomfortable. You'll need to rethink your work identity. But it's what unlocks your ability to be proactive, rather than reactive, and to point out opportunities to improve that nobody else sees. If you're not uncomfortable, you probably aren't growing.”

Timo Dechau - Founder at Deepskydata. Author of the How to fix your tracking eBook, writer of timo's newsletter:

“I would say - that I need to focus. I am too curious, so naturally, I dig into 100 topics at the same time.

What it changed - I picked one topic - tracking setup - and spent my energy on investigating it as far as possible. This brought me to a framework for tracking design and gave me plenty of great feedback when I wrote about it.

I stick to this now, that I try to pick one central topic and write about it. It does not always work out 😁”

Ergest Xheblati - Data Architect @Alvarez & Marsal, Author of MVSQL Patterns book, author of Data Patterns newsletter:

“I remember this one very well. I had joined a tech company and I wasn’t sure what direction to take in my career. I had been working with SQL and doing analytics engineering for a while and I had also worked on quantitative analytics.

I somehow had this notion that I needed to work on quantitative stuff in order to be closer to the business despite the fact that I didn’t enjoy it much. I enjoyed the engineering aspects much more.

I asked a director-level colleague about what he thought I should do and he plainly said: “Do what you enjoy and are really good at. You also happen to be very close to the business through the work that you do so why change careers? The business also happens to value your work as is since they’re already paying you for it.”

That advice was crucial to helping me continue to build my technical skills and eventually led to me writing my SQL patterns book.”

Avinash Kaushik - Chief Strategy Officer @Human Made Machine, ex-Chief Strategy Officer, BOD at Croud, ex-Google, Best-selling author, writer of the Occam's Razor newsletter:

“Every cutting-edge thing you learned in college will be stale in three years. Hence, take jobs where gaining new knowledge is a key part of being successful.”

Looking back, it has been proven to be true again and again and again. I've done my best in every role to stay close to the real work, and that has forced me to keep learning new skills. The alternative fate is I become yet another director/VP, whose primary success is driven by an ability to suck up or play company politics well.”

Benn Stancil - Founder Mode, author of benn.substack:

“For me, the best advice I ever got was to find real problems that I cared about answering.

There are lots of tutorials out there that teach technical skills on toy problems and sample datasets. These problems might teach you a few techniques, but they won't make you a better analyst, because learning to be an analyst requires asking questions, seeing a result, being curious about what that result tells you, asking more questions, and continuing to dig until you uncover something truly interesting and useful. With real data, on real problems that you care about you'll do this naturally; your curiosity will draw you further in. On sample problems, you'll often stop when you get to the answer in the back of the book—which, of course, doesn't exist for most questions you'll want to answer.”

Chu-Cheng Hsieh - CTO at SHEIN, ex-CDO at Etsy, Advisory Board Member @Google, Head of Alexa Voice Recognition @Amazon:

“The best advice is to always ask oneself: "Do you want to be effective or do you want to be right?" This applies to both data and leadership.

Let's start with data.

This advice reminds me to take calculated risks because speed matters in business. Often the cost of having comprehensive data to make the right decision requires lots of work. For example, one can set the p-value threshold at 0.01 but it means that you need to run A/B tests for an extended period of time – sometimes you probably never hit 0.01 because when you extend the experiment period, you have to also consider seasonality, model drifting, etc. And that's why most companies set p=0.05. Even if such a threshold naturally introduces false positive treatment, the choice is much more effective.

Let's talk about leadership. This question can be applied in different contexts. I'm providing one example here. Often we are facing multi-choice options. Say, there's almost no right answer in designing an organization. When I have to introduce an org change, I often favor consulting only key people (senior leaders, my manager, HRBP, etc.) While it's absolutely possible to consult every manager in the org to gather more information, keeping the group small comes with benefits like speed and alignment. It would take months if dozens of managers are involved in the process. On the contrary, it would be a blind decision to make an org decision without considering different perspectives. The question reminds me to seek a compromise between gathering inputs, achieving alignment, and making timely decisions.”

Alex Justman - Director, Product Management, ex-Vice President, Digital Resiliency at First Republic Bank:

“The best lesson I got from possibly one of the worst managers I ever had “Never be victimized by your circumstances”

You have to own your situation. If things turn against you you need to figure out how to assess the position you are in, and identify who you need to work with to provide a solution to your customers!”

Adam Kinney - VP Data and Analytics @Cantina Labs, ex-VP of Analytics at Mixpanel, ex Head of Advanced Analytics @Twitter, @Google:

“The best advice I got was to always ask questions about any request for data, metrics, analysis, etc, until I understand the ultimate decision that the requester is trying to make. All requests like these should ultimately be informing some decision, whether that it is a big one-off decision for the company or an ongoing decision, like metrics that inform a manager on whether their team's execution is on track. When you understand the decisions driving requests, you have an opportunity to figure out how best to inform the decision beyond the specific request and how to formulate it into the format that would be most effective for the decision-makers. It also helps with prioritization since the underlying decisions can usually be sorted for importance and urgency better than specific data requests.

The impact of doing this in my career has been that I moved from a tactical bit player with limited impact to someone who is driving strategy and has huge impact. It has also helped me coach my teams on how to have a bigger impact, which is generally a difficult thing for even pretty experienced data analysts to figure out.”

Read more - What Is the Best Advice You Have Ever Received?

Thanks for reading, and thank you to everyone who shared their learnings ⭐.

• Forecasting in Analytics: Choosing the Right Approach - How to predict revenue, user growth, and key business metrics using moving averages, regression, and ML.

• How Superset Stacks Up in Modern Analytics - A deep dive into Apache Superset and Preset - how they compare to traditional BI tools and why they stand out.

• How To Pull Stories From Data and Make Them Actionable - Methods and techniques that help transform data into clear and actionable insights. Tips on how to find a story or pattern in the data mess, what to do if you can’t find one, and how to present it in a way that is actionable.

  Subscribe now

Today, I want to bring back one of the very first interviews I ever did for my newsletter - with Reforge founder Brian Balfour. Our conversation took place about 4 years ago, but it was incredibly ahead of its time. I still find myself referring back to it often, especially now, as more and more analysts enter the market every day.

Just this week alone, I received 12 emails asking how to land a first project, find that first job, and stand out on a team. So, I decided to resurface one of my favorite interviews. Brian shares his perspective on analytics and offers timeless advice on how to grow and differentiate yourself in the field. It helped me back then, I hope it helps you, too.

Before we jump into it, a quick reminder:

Data Council is coming up on April 22-24 in Oakland, California.

It’s one of the most well-known events in the data world, bringing together over 1,400 attendees and 100+ speakers across all domains of data and analytics. I’ve got a 20% discount code to share with readers - use DAJ20 to save on your ticket. Hope to see you there!

Also, 2 other big events are happening this month:

• Analytics and Data Summit - April 8-10, Redwood Shores, California. The 3-day summit is organized and managed by the Oracle community and offers working sessions, hands-on labs, and all things Oracle.

• Tableau Conference - April 15-17, San Diego. Annual Tableau conference on dashboard, insights, and not agents. This is the time when they also announce their famous Iron Viz Champion.

You probably already know I’m a fan of Reforge. I took a Reforge course about 7 years ago, and it led me down a very different career path in product analytics. I was impressed by the nuance, detail, real-world examples they bring into their programs, and the level of expertise is the highest I’ve ever seen. Reforge absolutely stands out in material quality compared to any other product development course out there.

It’s a bit unfortunate that they don’t focus more on analytics - most of their programs are geared toward product managers, helping them build product intuition, strategy, and foundational thinking.

About 4 years ago, when I had just launched my newsletter, I cold-emailed Brian Balfour requesting an interview. I was just getting started - no large subscriber count or anything. To my surprise, Brian responded and agreed. We connected on a call and talked about analytics and career growth. It was such a pleasure speaking with him, and as expected, he has this unique ability to be crystal clear, concise, and right to the point. Of course, I will remain an advocate for Reforge regardless of how many qualitative tools they acquire.

I’m reposting our interview below. It’s short and timeless, and I hope it helps you choose your path in analytics and discover your own area of expertise.

Brian Balfour - Learn Your Domain and Differentiate Yourself

1. What qualities must a good data analyst have?

Different companies put various meanings for this role - data analyst, product analyst, data scientist, and they often overlap. A couple of qualities set apart from the average ones:

• The ability to explain and communicate a function, how to put data into a story. How do you communicate something you spent so many hours, years in your career to someone who doesn't know much?

• Don't take a question or a request on a surface, but discover, shape, and define real problems.

• Ability to merge product, market, or other knowledge with their day to day work. Upfront problem discovery.

And of course, everything else is very important - technical skills, must have curiosity.

2. What are some common mistakes that you notice data analysts sometimes make?

First, understand the project. What is the decision you are trying to make, and what is the impact of this decision? Is it small or medium? I often see over analysing, under analysing.

Your job as an individual contributor is to go out and gain the information you need to do your job - get the right insight, knowledge. If you feel you are not included in the discussions, you have to earn it. And, the way you earn your way into those conversations is by showing the ability to ask the right questions, provide the right insights, understand the context, connecting all pieces together, etc. Strong communication is very important. Be connected with other teams, because you can’t know every little piece of information or knowledge.

... and for that, you have to develop partnerships and work with multiple stakeholders or teams to access the right information.

I think the best organizational structure (or the hub), once you get to the right size, should follow a hybrid model where there is a centralized data team that is managing infrastructure, systems, applications, and then there is some form of an analyst sitting on a cross-functional product or marketing team. You can do it when (1) a company is at a certain stage and (2) they actually take data seriously enough to invest in it. It is still a rare thing these days... Most organizations do not have that. They are either running a very small centralized model or there is an analyst sitting on a random team.

Ideally, you are part of a truly cross-functional team working alongside product managers, engineers, designers, and not an order taker when a question or request comes up.

3. I think a Product Analyst is the most challenging role. It lives at the intersection between product, growth, and marketing. There are high expectations for analysts to comfortably navigate between all these domains and know their metrics and specifics and have enough statistical knowledge to back up their testing, monitoring, analysis, and insights. How can someone navigate all of this and not get lost?

These expectations are high for marketing or product managers as well. PMs need to know all the core products managing stuff, growth, and datasets. This is a common problem across all functions.

In a sense, PMs and Data Analysts are two probably the least specialized functions. If you are looking at Design, for example, designers went through this wave of specialization. There are now Product Designers, UX, UI, Content Designers, Strategist, Brand Designer, etc... And, there is only one type of Product Manager. There is now a growing specialization around Growth PM, for example. It seems like there is a little that specialization going on for Product Analyst, and maybe some others as well. My hypothesis is that these functions will grow over time.

I think it also sets a trap you are getting into when people feel like they have to be experts at everything. It will take a lifetime for you to learn everything. It’s an impossible task. My encouragement would be to pick a path:

1. Try to find a way to make it specialized in one area - Marketing Analyst, Growth Analyst, Financial Analyst, and understand the analyst function from that perspective and that domain. The depth you build there for your career will push you into a much more interesting position.

2. Many people pick a generalist side and do a little bit of each. The problem with that is it’s difficult to go deep, and as a result, you end up looking like all other analysts. Career management is Economics 101 - supply-demand. How would you put yourself in a situation where you are very low supply, and there is very large demand. And that’s how you get the roles and the job opportunities faster than others do.

It’s like a product strategy - how you differentiate yourself. The fastest way to differentiate yourself in your career is to go very deep. This is common advice I give to not just this function, but others as well.

4. What values do you try to foster within your team?

I’ll answer your question in a slightly different way. I read a lot of these blog posts about the XYZ values and principles a team should have. I think over time what I learned throughout my career is that values and principles are meaningless unless they are specific to your team, product, and mission.

A good example of this is the story from Fareed Mosavat (a Reforge EIR and was formerly a Director of Product at Slack). Many growth teams used to move and iterate very fast, building tons of Minimum Viable tests, and launching them in an extremely scrappy phase. At Slack, one of the core principles was Craftsmanship. It helped the company become successful in the early days. Fareed developed a principle on his team that merged the concept of Learning Fast with Craftsmanship. The tradeoff was that the tests they run had high design fidelity, which means that they would run fewer tests by putting more work into user research. This is an example of taking something that is important and molding it in the right situation.

Just copying and pasting generic values from another company isn’t going to work. You can use them as good inspiration points, but you really need to answer questions like who is your target audience, what is your differentiating strategy, what are the core behaviors, and things we need to do inside our company to achieve and execute that strategy. By the end of the day, if you make them more unique, they will mean more to you and your team.

Thanks for reading, everyone!

Related publications:

1. What Is the Best Advice You Have Ever Received?

2. Advancing Your Career in Data and Analytics | Peter Fishman

3. Building BI: Behind The Scenes | Benn Stancil

4. Why Today Is The Perfect Time to Learn Data | Seattle Data Guy

5. Navigating Data Tensions and the Future of Analytics | Lauren Balik

6. My Consulting Journey: $0 to $685K in 6 months

7. How to pass a first-round interview for data science roles

8. How To Hire Exceptional Analysts and Data Scientists

9. The Preparation Guide for Data Science and Analytics Interviews

PowerPoint presentations are one of the reasons we dread our jobs. No one enjoys making presentations. Unfortunately, the more senior and experienced we become, the more decks we have to create. There’s no escaping it.

In analytics, presentations are the most common output of our work. While they should (and must) be supplemented with a notebook, Excel, or a dashboard, decks remain the primary format for data storytelling when communicating with executives, the board, and stakeholders. We create board decks, monthly and weekly KPI reports, deep dives, investigations, roadmaps, proposals, A/B test readouts, retros, and all types of analyses. I typically have to create around 8–10 decks per month, and I struggle with each one. That’s how sad my life is.

So today, I’ll cover the dos and don’ts of creating decks, best practices, and common mistakes analysts make. Some of these are foundational data storytelling principles that we must follow, whether we like them or not, while others are personal learnings and recommendations.

Presentations, as an output of our work, effectively reflect our seniority and experience. The more junior a person is, the more slides, data points, narratives, and context they include in a deck, often making it overwhelming and unreadable.

It takes so much experience and knowledge to cram a month of work and insights into one small slide. Only one slide to illustrate correlation analysis, Y/Y user growth, seasonality patterns, revenue forecasting, or other projects you spent a lifetime on. Developing this skill requires training and practice, and 99% of analysts, including myself, struggle with it.

That’s why many companies still require candidates to create a deck as a take-home exercise when hiring. You can’t fake expertise with a deck. Your experience is reflected in the title, subject lines, colors, number of words used, number of slides, charts, references, etc. AI can only take us so far, and most AI-generated decks I’ve received are cringy and miss the mark.

So, what’s the best way to condense one slide in a way that would be easy to read, simple, and tell it all - the story, the context, the pattern, and the impact. How can one do it?

Learning from the Best

My biggest weakness is data storytelling. I’ve noticed that this is the case for many analysts. We focus so much on data and nuances that, naturally, every detail feels essential, every stat is important, and deciding what to “cut” or “keep” is never easy.

To make matters worse, I personally enjoy bold colors and prominent legends. I still don’t understand what’s wrong with having a bright orange bar next to a royal blue bar in a vertical chart with pink, bold, overlaying legends 🔥. Apparently, this is considered tasteless and distractive:

That’s why, if you’ve noticed, my newsletter includes over 50 publications dedicated to charts and visualizations - to override all my instincts. It’s also why I am always on the lookout for effective storytelling practices - I don’t trust myself one bit.

My list of experts on data storytelling:

The recommendations I share below are adopted from analysts, visualization experts, and presentation coaches:

• For KPI reports, board decks, or financial data, I use Bill Binch reports. Bill is an executive coach and board advisor who has spent his career in board meetings, so I use his examples when creating decks for the board and executives.

• When creating charts for the deck, I adopt methods shared by Adam Frost and Tobias Sturt, who teach Guardian Masterclasses in data visualization and share their practice at must-read AddTwo.

• For revenue data, subscriptions, and SaaS analytics, I adopt Christoph Janz’s frameworks and examples. Christoph is an investor, advisor, blogger, and famous SaaS expert. If you want to learn more about SaaS analytics and reporting, you should definitely read his articles. Christoph is a Managing Partner at Point Nine Capital and is behind such products as ChartMogul and Brightback - both famous subscription analytics platforms.

• For the deck structure and design, I follow Andrew Abela, founder of the ExtremePresentationMethod and author of Advanced Presentations by Design: Creating Communication That Drives Action. His publications are spot-on charts, slides, decks, dashboards, etc.

• Thomas Schmidt, a fellow data scientist, has shared the best practices of data storytelling from Metabase, Shopify, and DeepL in his recent guest post, How to Communicate Data Effectively.

• Lastly, I borrow slide examples from my favorite data storytelling newsletter, Chartr, which was run by a small team of 7 people before being acquired by Sherwood Media last year. It still remains the best data visualization media product out there.

Processing all of this, here are my key takeaways for crafting presentation decks.

Recommended practices for writing presentations:

Read more

• Predicting LTV with ML - Three case studies on using ML models to predict LTV for freemium. Learn how to choose the right model, incorporate seasonality effects, and address the specifics of LTV modeling in freemium.

• The Database of Winning A/B Tests - A curated collection of proven A/B tests, including success rates at different companies and the importance of transparent experimentation.

• Handling Missing Data: Should You Drop or Impute? - Exploratory Data Analysis: Techniques and methods for handling NULL values in modeling and analysis.

  Subscribe now

Analysts come in all forms and types - some are more advanced in statistics, some in data mining, and others - in business analytics - but what unites us all is the art of data storytelling.

It’s okay if you’re not an expert in Python, don’t fully understand data structures, or are slow with dashboarding. Trust me, your career success will depend on your ability to communicate - delivering concise, clear, and actionable insights.

It’s challenging to distill months of work with an ocean of context into a short sentence on a slide and a tiny graph. How can someone achieve it? 

While I am in Vancouver this week (and if you’re around and need a reminder of why analytics is awesome - let’s meet!), I turned to a fellow data scientist, Thomas Schmidt, to help us. Thomas, who also gave a talk on improving communication between data experts and stakeholders for the Munich Data Geeks, the largest data community in Germany, shares with us today the best practices of data storytelling that he developed at Metabase, Shopify, and DeepL. He has put together one of the most practical guides on communication in analytics I have ever seen, and I am so excited to publish it in my newsletter! 

Thomas is an analytics engineer at Metabase, an open-source BI tool that lets everyone work with data, with or without SQL, for internal and customer-facing embedded analytics. I use Metabase, and I find it intuitive and easy to start with. I also love their community stories - quick one-pagers focused on the fundamentals (I even published the A/B testing Checklist there a few years ago).

Below, Thomas will walk us through strategies and guides for bridging the gap between data and decision-making with clear, actionable communication.

TL;DR: How to Communicate Data Effectively to Executives

• Understand the core problem and tailor your message to your audience's needs.

• Start with a TL;DR: Summarize key insights upfront to respect executives' time.

• Keep it concise: Remove unnecessary details and clutter from presentations.

• Use colors intentionally: Highlight key points and create visual connections.

• Make bold statements: Provide confident recommendations, using appendices for details.

→ Apply these guides in your next presentation to ensure your data insights drive informed decisions.

Intro

Have you ever been to a foreign country trying to get directions from someone who doesn’t speak your language? That’s how I felt the first time I presented my complex data analysis to an executive team. Their confused looks made me realize that while the data made perfect sense to me, it was like speaking a foreign language to them. Just like a translator helps people understand each other, we need to bridge the gap between our detailed analyses and the decisions executives need to make. Effective communication isn’t just about sharing insights; it’s about making sure those insights are understood and lead to action. In this post, I’ll share lessons I’ve learned at Metabase, Shopify, and DeepL in turning data into clear, compelling messages that resonate with executives and drive informed decision-making.

Identify the real problem and tailor your message to the audience's needs

In our data jobs, we are often driven by curiosity and the joy of solving problems using data. However, it's crucial to understand two key aspects before diving into analytics projects: the core problem and the audience.

Understand the Core Problem

Stakeholders often approach us with a proposed solution in mind (“I need data X”). By digging into the 'why' and understanding the core problem behind the request, we can choose a better approach that serves their needs, leading to better project outcomes.

Know your audience

We don’t want to find ourselves in a situation where our audience feels like we are speaking a foreign language to them. Executives haven't been along for the entire analytical journey that we went through. Thus, we need to translate our findings for them. Before crafting a communication:

1. Identify what's important to them and their current priorities.
   Usually, this means strategic insights and actionable recommendations are preferred over details of the fancy ML algorithm you used for the project.

2. What do we need to explain, and what do they already know?
   Executives usually operate on a more abstract level and don’t necessarily know (and care) about each tiny detail of a process. Speak your audiences’ (domain) language! Explain acronyms, or don’t use them at all. Not everyone knows what an “ICP” (Ideal Customer Profile) is, and depending on the domain, it might even mean different things. 

3. How do we start our narrative to get folks excited?
   Talking to the head of sales? If you start by telling them that this will give them insights on how to beat the competitor, you will have their attention.

4. What is the best presentation format?
   Each format has its pros and cons: A written document might be good for async communication, but folks will be “alone” while reading it. You cannot clarify things with your words compared to a live presentation where they can directly ask questions (but it might be hard to find an overlapping time slot with the executives). Be sure to be thoughtful about that.

5. Assess your relationship and level of trust.
   At Shopify, we used the concept of a trust battery. Make sure you know “how charged it is” - this will influence how folks perceive and hear you and how you need to adjust your communication.

Don’t try to communicate to too many people at once: If your audience is too broad, narrow it down to the decision-maker and tailor your communication to their needs.

Provide a concise summary (TL;DR) upfront and keep messages clear and direct.

Executives will love you when you start using TL;DRs.

They need quick access to essential insights to make informed decisions promptly. In the fast-paced world, they juggle multiple priorities and have limited time for detailed reports. This is where the concept of TL;DR (Too Long; Didn’t Read) becomes invaluable. I first learned about this concept while working at Shopify, where we used a similar approach to our communication (be it for presentations, reports, or even longer Slack messages).

How to Craft an Effective TL;DR

When crafting a powerful TL;DR, the focus should be on what is most relevant and important to the audience:

• Summarize core findings and recommendations in a few sentences.

• Use clear, straightforward language, avoiding jargon.

• Use formatting to highlight key points with bullet points and bold text.

• End with an actionable recommendation or provocative question.

• Always position the TL;DR at the very beginning to capture the audience’s attention immediately.

Pro Tip: Add links to specific slides or sections with detailed data for those who want to dive deeper.

Example of a Well-Crafted TL;DR

Example: Improving Customer Service Response Times

Faster response times lead to higher customer satisfaction.

• Implementing the new ticketing system reduced average response time by 50%.

• Customer satisfaction scores increased by 20% after the change.

→ Recommendation: Expand the system to all customer service teams.

If you want to read more about TL;DRs, find some additional instructions in this article.

Time is Everything, and Less is More

As we learned in the TL;DR section, attention is a limited resource in our day-to-day world, and we need to fight for it. Here are some tips that have worked well for me in the past, along with good and bad examples.

1. Respect your Colleague's Time

Apart from using TL;DRs, make sure to use concise and direct communication. Avoid “bullshit” and unnecessary information (here is a good book on that topic), and aim to place your main message in the headline of your messages, slides, or report sections.  

Use concise and direct communication. Avoid unnecessary information and place your main message in the headline.

Example: Slack Message

The 2nd message lets you parse the information quicker and decide whether it is relevant and you want to dig deeper (i.e., continue reading in the thread). There are a lot of great additional examples in the book “writing without bullshit” in case you want to read further.

2. Simplify slides and reports by reducing text and eliminating non-essential elements.

The above tips connect well with the next topic we are going to cover: Clutter.

In the Slack message example above, you saw a lot of additional information that did not support the message:

• “I listened [...] on my Podcast Addict app […] I made 40km that day” → Your target audience will not care! Make a post in the #biking channel or tell your friends instead if you need to. 

• “It is like an hour long but I can totally recommend it to you” → If someone decides to check out the link, they will figure that out.

These are extreme examples, but we tend to add “clutter” to our presentations or reports as well. When was the last time you listened to a presentation and started thinking about something else since you could not follow all the information on that single slide? In those cases, the presenter lost your attention because of cognitive overload. Decluttering can help reduce the cognitive load for your audience and make it easier to parse your presentation or report. 

Things you can start doing today:

• Reduce the amount of text: When you give a presentation, your slides don’t necessarily need a lot of text. You can guide and explain verbally.

• Challenge each element in your chart: Look at each of your axes, labels, data points, grid lines, etc., and ask yourself: Would eliminating this element change anything? If the answer is no, get rid of it.

• Avoid gifs or animations: If they are not helpful for your message, avoid them. Our brains get distracted by such content very quickly. Only use it if there is a very good reason.

Here is an example of 2 charts that visualize the exact same data using different amounts of clutter:

I have no doubt that you will notice how much heavier the first chart feels. How long would it have taken you to extract the message? No executive will have time for that!

If you use a BI tool, you will usually find options to reduce clutter there.
Here is another example that uses Metabase. It mostly comes with reasonable defaults, but we can still tweak it a bit:

Labeling the bars directly and removing the y-axis ticks, makes the chart easier to process. Readers can directly compare the numbers for the jump in AOV (Average Order Value) without needing to jump back and forth between the axis and the bars. We also intentionally rounded the values to remove decimal precision since it is not needed for our message.

3. Apply colors consistently to create visual connections and highlight key points.

Colors are very powerful in our communication toolbox - if used wisely and consistently. 

Make sure to be intentional about your use of color. One of my favorite examples is to use it for visual connections between slide titles and my charts, where I try to color a category I mention in the same way it would appear in the chart. Additionally, you can use black vs. gray color and bold vs. normal text to create a visual hierarchy. This helps guide the consumer through the content. Here is an example:

You see how the blue and orange colors connect to “increase” or “decrease” respectively. The hue makes the important parts stand out while we push other things to the background for context. This helps to reduce the cognitive load and allows your colleagues to process the content in a shorter time while increasing the chance that they perceive your message. This can also help across slides: If you stay with your color palette, your audience will quickly realize it, and it can be powerful to connect the dots with your previous story.

Let’s apply this to our Metabase example from above:

Adding a clear indication of when the Shop redesign happened, which led to the increased AOV, can help us make our story stand out.

4. Give clear, confident recommendations and use appendices for details

“Don’t caveat everything”

Once, an executive at Shopify told me that data scientists tend to caveat everything, and he often wished that folks were bolder about their recommendations. While it is natural for data professionals to want to highlight the nuances and limitations of our analysis, it is important to remember that executives are looking for clear, actionable insights. They prefer confident recommendations to make strategic decisions quickly over being walked through the details of your analysis.

Some examples:

• Caveated Statement: "If the proposed customer retention strategy is implemented and the market response is favorable, we might see a potential decrease in customer churn by up to 5%."

• Bold Statement: "Implementing the proposed customer retention strategy will decrease customer churn by 5%."

• Caveated Statement: "While there are multiple factors to consider, and it’s possible that some variables might change, we believe that the new feature could potentially improve user engagement."

• Bold Statement: "The new feature will improve user engagement."

You might argue that there are cases where it is important to add caveats. A good approach in those cases is to have an appendix and refer to it. This might give you more confidence, and executives will have the chance to dig deeper if needed.

Recap

Effective communication with executives requires thoughtful consideration of their time and needs. By understanding the problem and your audience, starting with a TL;DR, removing clutter, using colors wisely, and making bold statements, you can ensure your insights are understood and lead to actionable decisions.

Thanks, Thomas!

• Find and connect with Thomas on LinkedIn and X.

• Learn more about Metabase.

• Check their resources and guides.

Thanks for reading, everyone!

This month, paid subscribers learned about:

• Ranking The Top Used Product Features - How to locate the most frequently used product features to learn what drives the highest user engagement - steps and SQL.

• How To Make A Sandwich in 587 Steps - because it’s my newsletter, so I can disagree about data platforms.

• WWDC 2024 Recap: Top Announcements Impacting Analytics - What you need to know about recent key updates from Apple and how they impact mobile analytics.

  Subscribe now

When I launched my newsletter a few years ago, there weren't many bloggers writing about analytics, unlike today. Substack now lists more than 360 newsletters about analytics. However, only a few writers truly stand out, one of whom is .

For many years, has embraced the classic, foundational themes of data analytics and BI. For this reason, I love his blog because, truly, there is nothing more impactful or exciting than analytics.

Today marks a historical milestone for my humble newsletter - I am excited to share my interview with my favorite blogger and analyst, the co-founder of Mode, Benn Stancil! ⭐

In our conversation, we discuss the tricky nature of BI, the pitfalls of building data analytics tools, new trends in analytics, and more.  

What is the most challenging aspect of building BI tools?

You have to be really disciplined about identifying who your customer is and focus on selling to them. 

This is why BI is tricky: it’s easy to end up selling to 3-4 different people. Especially early on when you are looking for customers, it's very tempting to drift away from your primary persona.

For Mode, we built a tool primarily for analysts. However, analysts are serving different stakeholders, and then there is also admin and management, so it is actually a tool that was meant to serve one group of people but consumed by another, and then, there are different categories of users. Though we mostly were always more focused on data teams than these other groups, you inevitability find yourself questioning how much you should build features for them, too.

If I do it again, I’d basically try to recognize from the beginning whom we are serving and be very disciplined about it: “This is the product for just this group of people.” Stick with this persona, make something that's as good for them as possible, and don't worry too much about other types of buyers

Other successful data tools do this pretty well. R Studio and Jupyter Notebooks, for example, mostly don’t deviate from being technical tools. And tools like Mixpanel know where to draw their boundaries - e.g., primary for product managers. 

It’s hard to do this with BI, because they can get used by so many people, but it’s necessary.

You wrote some time ago something that stuck with me for years: "No matter how much a visualization technology can do, people will want more...Give people a line chart, and they’ll want to add another line on a second axis. Then they’ll want to turn one of the lines into a bar chart; the bar chart will need multiple series; first stacked, then grouped. If the chart is a time series, they’ll then want to group data by different intervals: by day, by week, by month. But do weeks start on Sunday or Monday? They’ll want to choose. Which time zone are dates grouped in? Can you exclude weekends? Can you treat incomplete periods differently?" 

Agree, but how do you navigate all these requests and directions when building a BI tool?

This is one of the reasons BI is hard. There's no easy way out of these requests—it's just about effort.

Let’s take Twitter, they released one killer feature that worked really well, and people loved it. However, for BI, I think there is no one killer feature, people need to have the ability to do many things. People need more. So, the mode for every BI company is to put effort into building a thousand of features. 

Looker had a clever thing with semantic layers and ML, or Tableau, for example, focused on visualizations, and there are many companies who try to be clever at one or another, but for the most part, to be a really big and last as a BI tool visualization tool, you need to support many different things - integrations, alerts, different dashboards formatting, different ways to consume data, permissions, management, different ways to distribute things, … and it’s a very very long list. 

The trap for many companies who want to get into the BI space is the idea that we can build 20% of Power BI and cover 80% of usage, and it will be enough. Some use the 80/20 rule (you can get 80% of the experience by building 20% of the features), but the thing is, you can’t do something 20% and think it’s enough. You need to build 100% for that 80%. Otherwise, you will end up with a very frustrating product. With BI, it’s really hard to build this 80%. That’s the trap.

What’s your favorite feature that was never shipped?

Tons.

We always wanted to build internal Quora - a place where people can ask questions, like a forum. A resource to find answers for questions, a place where conversation would live. At Mode, every report was like a snapshot in a way, and someone could ask a question, but the answer is fixed in time to the question they saw. I don’t know if this would really work, but I would love to see it.

When I was at Yammer, which was a product similar to Workplace (aka, Facebook for work), This was how the data team handled answering questions, and it worked really well.

I was thinking of a Version Control for analysts. It’s a must-have and is currently absent in today’s data landscape.

Yeah, I am surprised that there is no Asana or Linear for analysts yet, given how much exposure data teams have gotten over the last few years. 

It’d probably have to be attached to another tool you are using, though. It’s hard to get right.

Which BI features are difficult to build?

Visualizations are hard. People always want to do new stuff with it. It’s complicated, takes effort. 

Also, Version Control, but not code, but rather who-solved-what-when. For example, if I share a link of a report - what am I sharing? Is it a snapshot, when is it updated, parameters, what we do about it, etc.. 

Another thing, which is related, there is no sense of production. Unlike in engineering, where it’s clear the scope you are maintaining, there is a sandbox, staging, etc.. Everything in data is half-of-production. If someone asks me a question, and I send a link in Slack to a chart - is that production? Kind of. It was in a moment when I shared it. It’s production in the sense that it was sent by someone on the data team, which is a kind of implicit confirmation that it works… People want to trust BI, and trust it’s verified. But it’s impossible to know who maintains it, or if it’s maintained at all. This would be nice, but very hard to do.

The best versions of this that I've seen are when companies have relatively few dashboards. They say "We'll maintain these exact things, and that's it." If you need more, come talk to us.

We were trying to expand the self-serve realm - if you need more - talk to us or bring to the mechanic.    

Over 10 years ago, we had bad tools, and most of the work was done manually. Now we have this amazing tech, yet the time to value hasn’t changed. The productivity output didn’t change. Why do you think this is the case?

We've made it much easier to build dashboards, which is great, but having more data doesn't mean that we know what to do with it. 

For example, take a car dashboard - no information there will make you a better driver. The same is true with BI. A few basic numbers is often more useful than data on everything. People don’t know what to do with that data.

This is roughly what we have done - move away from thinking about how to be a better driver to thinking about how to put more metrics on the dashboard. 

What new directions in BI tooling do you see today? Do you agree with or like new trends?

Different things for sure will catch. 10 years ago when notebooks became popular, it was nice. Same with Tableau. Gen AI could make tools easier to work with too. For example, the Tableau interface is hard. There is a learning curve to using any BI tool. GenAI could be a way to put a conversational interface on top of BI. Even if it’s not AI agents solving all our problems, it could be a more accessible interface on top of BI.

If I have to pick what would really change things - and I don’t think I’m optimistic about it ever existing - it’d be someone making Excel work at scale. There are some attempts, like Sigma, though these are more spreadsheet interfaces on top of databases. But that’s not really Excel because charts and reports are a representation of data that lives in a database. The genius of Excel is that it’s both - the representation and data itself. It’s a file, which is a very clean concept to understand and not just a view. 

Like Equals? 

Equals is interesting. They started as a Google sheet competitor. I don’t know if that quite replaces it, but for sure it gets closer. But the direction they have gone is more BI. Now they offer dashboards and other things. I am afraid of it becoming a BI tool with a spreadsheet backend instead of a spreadsheet tool.

It’s similar to Hex. It’s a very good notebook. And initially, they were focused on being a notebook tool. But now, they position themselves more as BI, and the notebook is the backend for the BI tool. It’s notebook-based BI, rather than a notebook. This is the drift that makes BI so hard - it’s easy to evolve from building a great, specialized tool into building a much more general BI tool with a twist.

Is there anything else you want to share to encourage or inspire people to learn data?

Don’t do it for the data alone. Do it for the problem you want to solve. 

It’s easy to say here is a list of technical things I want to learn, but my question would be - do you want to learn technical things or do the work those technical things are meant to help you do? 

The time I had the most fun - was when you work on a problem you want to solve. Get to the bottom of it. By solving these problems.

Thank you, Benn!

Find Benn:

• LinkedIn

• X 

• Benn substack

My favorite publications from Benn's blog: 

1. The third rail

2. The missing piece of the modern data stack

3. Is BI dead?

4. LLMs shouldn’t write SQL

5. Why do people want to be analytics engineers?

6. The end of Big Data

7. Disband the analytics team

8. Has SQL gone too far?

9. The rise of the analytics pretendgineer

10. How analysis dies

This month, paid subscribers learned about:

• Data Analyst Toolbox: Must-have tools and applications for data analysis and data science that can save you time and effort. Sharing my personal analytics toolkit.

• Methods To Accelerate A/B Testing: Strategies and statistical methods to increase test velocity. ​​Learn what statistical methods and solutions you can leverage to speed up testing and how to balance trust and speed.

• Introduction To Proxy Metrics: A guide to identifying and implementing effective proxy metrics, with examples and use cases.

  Subscribe now

Measuring the impact of product feature optimizations is not easy.

Identifying what drives top-line conversions requires significant analytical effort and time: 

• More feature usage does not necessarily imply higher conversion rates.

• High conversion rates do not necessarily lead to improvement in retention.

• Improvements in retention may not translate into increased revenue.

It all comes down to understanding the causal relationships between metrics and user actions. 

To dive deeper into calculating and measuring cause-and-effect relationships, I asked Tom Laufer, CEO and Co-founder of Loops, to share with us the best practices and frameworks in analytics. 

I met Tom in San Francisco a few months ago. What was intended to be a brief introduction turned into an hours-long conversation about the state of data analytics, the challenges faced by analytics teams, and ways to navigate them. It was, by far, the most refreshing, intelligent, and direct discussion on analytics I've had in a while. I knew then that I had to ask Tom to write for me, and today, I am so excited to invite him to be a guest writer.

What’s even more exciting is that I am joining Tom next week in San Francisco at the Product Analytics Leader's Meetup, where we will discuss ways to build high-impact product teams, enhance insight-actionability, avoid pitfalls in defining KPIs, introduce causal analytics, and more. Hope to see everyone there!

Tom Laufer previously led growth and analytics at Google EMEA. Over a career centered on product and growth, he has served in both product management and data analytics. In 2020, he co-founded Loops, a product analytics tool designed to help data and team maximize their KPIs by identifying hidden actionable opportunities within their product data.

Today, Tom will share his perspective on understanding cause and effect and will walk us through 4 different analyses: Conversion drivers, Retention drivers, Engagement drivers, and Feature funnel analysis.

How to Size Feature-to-KPI Contribution 

As analysts, your focus is to identify what could drive your topline metrics. It’s clear you need to focus on the features, content, and journeys that drive your KPIs, but how should you do this?

The many data sources and correlations you have at hand can be very confusing. You might see that users of one feature, say “Search,” are more likely to convert. However, this does not necessarily mean you should focus on Search? While users that “Search” have some intent, this feature itself may have little impact on your other KPIs once they convert. It could be that the intent itself is the reason why these users convert or retain, and not because of the feature search. Meaning, it could be purely a correlation. That's why understanding correlation versus causation is so important.

Therefore, it’s important to understand the hidden causal relationships in your data. 

Cause and Effect Relationships Rule:

By identifying and analyzing the cause-and-effect relationships between different variables, you can gain a deeper understanding of what drives your product's growth and what factors are most critical to optimize. This helps you make more informed decisions, take the necessary steps to improve your product, and ultimately drive successful growth.

Going out on a limb here… if you understand the cause and effect behind your KPI changes, you can expect your analytics life to be better. The “knowing” you get to with causal analysis is far more fun than the “Guessing” - plus, you can achieve more with less work. You are much more likely to run successful experiments and to take correct action, than compared to running off of the pure correlations that you might have today. And… by being more successful through causal analysis, you will drive a real impact on company growth and gain well-deserved recognition.

An Actionable Analytic Framework:

The causal analyses I’m about to discuss each offer significant value for analytics and growth teams. They provide a framework for identifying the biggest opportunities to improve your KPIs. 

Also, each is achievable - you can begin implementing these analyses right away, either manually (read the post here on how to model intent with causal models) or through certain vendor solutions. In this example, the analyses are shown using the Loops platform, but you can learn how to build your own opportunity sizing models from my post on this topic, and in the “Building an Opportunity Sizing Model” section below.

Key info about the sample scenario used in this article:

In the example below, I use a photo/video editing app that offers Free and Paid versions. I will analyze the same features against different KPIs.

Conversion Drivers Analysis

• A causal inference model-based analysis on new users, 1st week conversion.

• 1st week conversion is typical for our example of a photo/video editing app.

• Only pre-conversion new user experiences are considered.

The table below shows a list of features that have the highest impact on getting users to convert within the 1st week. The “Split Video” feature has the highest potential to improve the conversion KPI, at 38.63%. What’s very interesting is that if you get all the users to adopt this feature your KPI will go from 21.1% to 38.63%, which represents 82.86% lift, as shown in green on the far right. 

The Growth Simulator below allows us to do what-if analysis (sensitivity analysis) to understand if we want to prioritize this opportunity.  We want to identify features with high impact and good upside in terms of adoption rate. For example, we see that the Clean Object and Replay features have relatively low adoption compared to others and represent potential growth opportunities. 

Retention Drivers Analysis

• Free Users

• 3rd Week Retention 

In this causal analysis, I focused on the 3rd Week Retention of free users. Even for free users, improving retention has merit and value - they may convert later, contribute to virality, and provide valuable data for causal analysis, in this case via AI models. 

The format of the table and graphic below is the same as shown above for the Conversion KPI, showing the Feature, Adoption Rate, Goal in Adopters, and Maximum Potential impact.

Interestingly, none of the “winners” from the new user conversion analysis above rank at the top of this list of high-impact features for 3rd Week Retention. Instead, the “Audio AI Fix” stands out with a maximum potential lift of 113.26%. Additionally, the features listed in rows 2 and 3 (“IAdd Text” and “Glitch Video Effect”) also show 90%+ maximum potential impact. All three features have low Current Adoption and high Goal in Adopters rates.

Engagement Drivers Analysis

• Paying Users

• Engagement is defined as the number of times the core value "export video/photo" is performed in a 28-day window. 

Unlike the other analysis, this one does not look at new users. It is focused on existing, paying users, previously activated and with a clear pattern of usage. 

The analysis looks at patterns of engagement and identifies the impact of adopting different features on the level of engagement (Exporting a Video/Photo over 28 days). The analysis identified multiple features with a positive impact on the Export action. In the table below, you can see the projected KPI lift in the number of “exports” in a 28-day window for key features based on various lifts in the adoption rate (the growth simulator enables you to play with different levels of intent). The “Magic Eraser“ feature potential lift in this engagement KPI is 14.42%, for example. The table is ranking the feature by the potential lift:

It is interesting that two of these features were not included in our top features for our new user conversion analysis, nor on our top features for 3rd week retention.  

This analysis highlights the difference that the user’s stage has on the impact of features on a KPI - i.e. the drivers for existing users that have already developed a habit around your product are different than for new users.  The insight informs our understanding of the user lifecycle and the decisions we make to optimize our KPIs through each stage. In the table below, “Magic Eraser” for example, can drive a 14.42% lift in the amount of Exports per 28-day window.

Feature Funnel Analysis

While we have found interesting opportunities from the previous analyses, this next step involves figuring out what to do with that information to increase the adoption of these features and journeys. 

The best way to look at this is through a Feature Funnel analysis, where the funnel for each feature is analyzed at its final stage. This analysis makes it very clear where new opportunities lie, and where existing efforts have had little result.

The table shows the key four measures of this analysis:

The table and the graphic below highlight the two extreme cases of opportunities in our photo/video editing app example.

Key Insights:

Having presented these different analyses, I’ve demonstrated that you can get to some really deep insights using causal analytics. These are the key insights to keep in mind:

• Features Rarely Fit Every Stage: The features that initially attract users to purchase the product are not necessarily the same features that get them to form a habit and engage more with your product later on. You need to tailor the right journeys based on the user’s maturity / stage, and the KPIs you are looking to move.

• Go deeper: To understand why users are not adopting certain features, you’ve got to dig deeper. Doing so will make your insight much more actionable.

• Optimize Your Free vs Paid Mix: Identify which features should be offered in the Free version versus those that should stay in the Paid version. Ensure you are not giving away capabilities for which users would willingly pay. 

  • Features that drive conversion can be promoted in the paywall, building awareness of them, then maybe offering them free for X days.

  • Features that drive engagement for paid users should probably be kept in the paid version. Promote them later in the user journey to increase engagement - which leads to lift retention, upsells, virality, etc. 

  • Features that drive free retention but don't add a lot for monetization should be kept in the free version and promoted there. 

  • Acknowledge the Value versus Frequency Trade-Off Examine the feature value/frequency balance. Don’t expect the sexy features that attracted your users to your platform and had high value for new users, to be the features they run most often down the line. By digging deep into your product usage data, you can identify these trends, enabling you to maximize KPIs at every stage of the user lifecycle.

How to Build an Opportunity-Sizing Model

The opportunity sizing model is derived from the causal insights models. While a complex analysis, opportunity modeling can help you discover deep and sometimes unexpected insights. Because you have goals to meet, you need to prove that the potential lift from features/journeys could significantly contribute to your KPIs. For example, growth teams run many initiatives that, even if they achieve their maximum potential, would not affect the numbers significantly. Conversely, other opportunities are much more significant than they would have ever thought. Causal insight helps minimize the risk of going off in the wrong direction. 

Here are the key elements of opportunity modeling - a high-level, simplified overview:

1. The specific reach: How many users will be exposed to the new feature/experience/launch? You can sometimes use a funnel to measure this, just ensure you only include the user segments that are relevant for the feature.

2. The potential impact on one core metric. Segmentation is crucial here since different segments might react differently to the tested opportunity. Compare users from similar campaigns with similar attributes. Grouping these users enables you to understand behavior better. 

3. The impact on business metrics: For example, the projected impact on revenue? Let’s assume you improve 2nd-week retention —> how would it affect your WAU or Revenue?

Most product analytics solutions are built as visualization tools - dashboards that might show a correlated feature but without any causal certainty. For the analyses reviewed in this article, you’ll want to build or find a solution that provides the estimated impact of each opportunity presented so you can decide on how you will prioritize them.

In Conclusion:

Causal analytics deliver robust insights and quantify the impact of each opportunity. Having these hypotheses, the ones that will win, is a complete paradigm shift for growth and analytics teams. With causal analytics, they can make a real impact on growth. 

And when growth happens, people get well-deserved recognition and yes …even promotions!

Thanks, Tom!

• Find and connect with Tom on LinkedIn and X.

• Learn more about Loops - You don't have insights; You have dashboards.

• Learn more about product analytics - Loops blog.

Thanks for reading, everyone. Until next week!

Subscribe now

First, a few quick announcements: 

I have two in-person events coming up next month:

1. June 4, San Francisco: Product Analytics Leader's Meetup - I’ll join product analytics leaders from Reforge, Google, Meta, Eventbrite, Pinterest, Dropbox, PayPal, and others to discuss ways to build high-impact product teams, avoid pitfalls in defining KPIs, explore causal analytics, and more!

2. June 6, San Francisco: Impact of Data Processing Tools on Analytics and Reporting - I will moderate a panel featuring data veterans such as Michel Tricot (CEO at Airbyte), ​Pete Hunt (CEO at Dagster Labs), ​David Krakov (CEO at Honeydew), Peter Fishman (CEO at Mozart Data), and celebrities like and . We’ll discuss ETL tooling, data quality KPIs, the future of ETL, semantic layers, and root cause analysis using metrics decision trees. With such honorable speakers rarely in town, this is an event you won’t want to miss!

This newsletter is a little special ⭐.

Once per year, I reach out to my network of analysts and data experts with one question: What is the best advice you received that changed your career? Today, I want to share some of the wonderful responses I received. I hope they will inspire all of us to grow in analytics and get empowered with data.

What Is the Best Advice You Have Ever Received?

- Data engineer and architect, author, global keynote speaker, professor, advisor, investor, podcaster, and writer of newsletter:

Careers don’t often make much sense. Mine certainly doesn’t. You’ll probably look back on your career and wonder how you got to where you are today. And you’ll be confused about where to go next. This is part of the journey.

Here’s some advice that helped me navigate the challenges and uncertainty in my career.

• Reputation matters. People have long memories, and the world is a small place.

• Success isn’t so much about being smarter than everyone else; it is about avoiding stupid things that will put your career in the graveyard. Lying, stealing, cheating, and nastiness are a few ways to kill your career. If you look at the graveyard of careers, you’ll find many supposedly smart people buried there. Don’t be one of them.

• Continuously learn. Read, talk to people, write, and explore new ideas and skills. Learning compounds over time. The person who continuously learns is a force of nature.

• My boss once told me, “When I ask for the time, don’t tell me how to make a watch. Just tell me the time.” He meant that I needed to give a direct and clear answer when he asked a question. Clear and concise communication is key.

• Lift others up. The more people you can help, the more they will help you when needed.

Lindsay Murphy - Head of Data at Secoda, advanced dbt instructor, host of Women Lead Data podcast:

“Data is not a technical role. It's a strategic role."

Data people tend to hyperfocus on things like pipelines, what technical skills to learn, and what's the coolest new tool everyone is using. But in reality, data is a strategic asset for a business. 

All of the work you do should be to understand and optimize the strategy of the business. Data people should think of themselves as strategic partners to business leaders, not technical partners.

The tech is the means to an end, but if you don't understand how your business functions, what strategic levers your company can pull to succeed, and how data helps them do that, then you're going to focus your limited resources on the wrong things, and have a difficult time being successful.

Chug Abramowitz - Co-Founder at New Agenda Group, ex-Growth Leadership @Spotify, @Belkin, @CurrentHealth, @vidIQ:

A mentor once told me, "overthinking decisions is a waste of time." He was referring to choosing between two jobs or whether to stay in NYC or move to LA and getting stuck in analysis paralysis. The key is to examine your options but then enthusiastically embrace your choice!  Most often, committing and giving your all to your choice is far more important to ensure an awesome outcome than the decision.

Carlos Aguilar - Founder/CEO at Hashboard, ex-VP Data @Flatiron Health, Advisor, ex-Amazon Robotics

I worked in supply chain for about five years and the best advice I received from an early manager at the time was to zoom way in and dig deep when troubleshooting a problem. My natural tendency at the time was to look at the data as a whole and miss the human element.

If there was a warehouse that was operating poorly, just go look and see what's going on at the warehouse. Go talk to people, inspect the shelves, look at the actual products. The best data people are able to operate at multiple levels: zooming way out and thinking about the data as a whole and also zooming way in and not losing empathy for what is happening on the ground.

The greatest thing about data is that it can scale empathy, so don't lose sight of the fact that each event could represent a person (or a product, etc.)

Shinji Kim - CEO/Founder at Select Star, ex-CEO of Concord Systems, advisor, investor, and mentor:

"Be yourself" is the best advice I received early in my career.

I had many ideas, thoughts, and questions about how things work but often felt self-conscious about whether I was asking the right questions, making the right decisions, or fitting into my environment. Getting comfortable with the fact that I may look or act differently than others gave me the confidence to push myself further and focus on my goals.

Ryan Janssen - CEO/Co-Founder at Zenlytic, entrepreneur, data scientist, and investor:

Have intentionality.

Early in my career, I asked my mentor to help me choose between two very different jobs. I had a big pro-con matrix and a financial spreadsheet. I talked his ear off for 10 minutes before he interrupted and asked "But how do you want to spend this next 5-year cycle of your career?" And I realized I couldn't answer him clearly.

Many people think they have a good idea what they want to get out of their career, but actually don't. Or they continue on a path that clearly lacks the trajectory they need to get there.

My advice is to take time to introspect. Schedule time if you have to. Write things down. Be deliberate and detailed about what you want to do in your career. Be deliberate about the most important projects you need to work on now to get there. Careers are short and it's easy to let time slip away; for satisfaction and success, have intentionality about your career every single day.

Reflecting on advice I received last year:

- Operating Partner at OpenView, the author of the newsletter: 

Look for ways to replace yourself. Early in your career, you get ahead by being the best analyst -- the person who knows the data better than anyone else. That quickly changes. To truly advance, you need to find ways to hand off your responsibilities to someone else. 

It'll be painful in the short term. You'll feel uncomfortable. You'll need to rethink your work identity. But it's what unlocks your ability to be proactive, rather than reactive, and to point out opportunities to improve that nobody else sees. If you're not uncomfortable, you probably aren't growing.

- Founder at Deepskydata. Author of the How to fix your tracking eBook, writer of newsletter:  

I would say - that I need to focus. I am too curious, so naturally, I dig into 100 topics at the same time.

What it changed - I picked one topic - tracking setup - and spent my energy on investigating it as far as possible. This brought me to a framework for tracking design and gave me plenty of great feedback when I wrote about it.

I stick to this now, that I try to pick one central topic and write about it. It does not always work out 😁

Chetan Sharma - CEO at Eppo, early data scientist at Airbnb, Webflow: 

Here's a piece of advice that was useful to me, from my good friend Erik Goldman, founding CTO of Vanta.

"When considering opportunities, reduce your decision space to one "got to have it" thing. Decision quality goes down when you simultaneously consider a composite of factors: wealth creation, technical development, adding well-known brands to your resume, people management opportunities. Far better to condense to one goal: "I want to learn about AI" or "I want to start a company after this". For example, before Eppo I knew I wanted to start a company, and thus indexed heavily on proximity to leadership, reporting directly to the CTO."

For the analyst-specific career path, my big recommendations are:

1. Write, both internally and externally. Give lots of presentations. Doing these will grow your brand and credibility, far more than painfully persuading business stakeholders one by one. Writing one external blog post at Airbnb has done more for my career than years of in-house IC work.

2. Think of analyst work as a stepping stone to another functional domain: product, eng, marketing. This isn't to say that you have to leave data work, but the highest impact analysts will be multi-brained with one of those.

Ergest Xheblati - Author of MVSQL Patterns book, author of newsletter:

I remember this one very well. I had joined a tech company and I wasn’t sure what direction to take in my career. I had been working with SQL and doing analytics engineering for a while and I had also worked on quantitative analytics.

I somehow had this notion that I needed to work on quantitative stuff in order to be closer to the business despite the fact that I didn’t enjoy it much. I enjoyed the engineering aspects much more.

I asked a director-level colleague about what he thought I should do and he plainly said: “Do what you enjoy and are really good at. You also happen to be very close to the business through the work that you do so why change careers? The business also happens to value your work as is since they’re already paying you for it.”

That advice was crucial to helping me continue to build my technical skills and eventually led to me writing my SQL patterns book.

- Co-founder and CPO at Hyperquery, ex @Airbnb, @Wayfair. Author of the newsletter:

The best advice I've gotten came from my skip manager at Wayfair, who told me:

"You should always try to work at places that you're really excited to work at." 

At the time, I remember thinking he was just trying to keep me at Wayfair, but in hindsight, I realized this was fantastic advice. When you enjoy your work, everything compounds. You learn faster, you advance faster, you're more creative. Plus, you're happier, which means more of the rest.”

- CEO&Co-founder @Gabe.ai, Investor, Advisor, Shift Operator @Data Quality Camp. Author of the newsletter:

Start with the problem, not the solution. Often times data teams get so wrapped up in complexity that they fail to step back and remember the work we are doing is not academic. If you always root work to clearly defining the problem, you will never have to worry about whether your projects deliver value or not.

Avinash Kaushik - Chief Strategy Officer, BOD at Croud, Best-selling author, writer of the Occam's Razor newsletter:

“Every cutting-edge thing you learned in college will be stale in three years. Hence, take jobs where gaining new knowledge is a key part of being successful.”

Looking back, it has been proven to be true again and again and again. I've done my best in every role to stay close to the real work, and that has forced me to keep learning new skills. The alternative fate is I become yet another director/VP, whose primary success is driven by an ability to suck up or play company politics well.”

- Founder Mode, author of :

For me, the best advice I ever got was to find real problems that I cared about answering. 

There are lots of tutorials out there that teach technical skills on toy problems and sample datasets. These problems might teach you a few techniques, but they won't make you a better analyst, because learning to be an analyst requires asking questions, seeing a result, being curious about what that result tells you, asking more questions, and continuing to dig until you uncover something truly interesting and useful. With real data, on real problems that you care about you'll do this naturally; your curiosity will draw you further in. On sample problems, you'll often stop when you get to the answer in the back of the book—which, of course, doesn't exist for most questions you'll want to answer.

Chu-Cheng Hsieh - CTO at SHEIN, ex-CDO at Etsy, Advisory Board Member @Google, Head of Alexa Voice Recognition @Amazon:

The best advice is to always ask oneself: "Do you want to be effective or do you want to be right?" This applies to both data and leadership.

Let's start with data.

This advice reminds me to take calculated risks because speed matters in business. Often the cost of having comprehensive data to make the right decision requires lots of work. For example, one can set the p-value threshold at 0.01 but it means that you need to run A/B tests for an extended period of time – sometimes you probably never hit 0.01 because when you extend the experiment period, you have to also consider seasonality, model drifting, etc. And that's why most companies set p=0.05. Even if such a threshold naturally introduces false positive treatment, the choice is much more effective.

Let's talk about leadership. This question can be applied in different contexts. I'm providing one example here. Often we are facing multi-choice options. Say, there's almost no right answer in designing an organization. When I have to introduce an org change, I often favor consulting only key people (senior leaders, my manager, HRBP, etc.) While it's absolutely possible to consult every manager in the org to gather more information, keeping the group small comes with benefits like speed and alignment. It would take months if dozens of managers are involved in the process. On the contrary, it would be a blind decision to make an org decision without considering different perspectives. The question reminds me to seek a compromise between gathering inputs, achieving alignment, and making timely decisions.

Adam Kinney - VP of Analytics at Mixpanel, ex Head of Advanced Analytics @Twitter, @Google:

The best advice I got was to always ask questions about any request for data, metrics, analysis, etc, until I understand the ultimate decision that the requester is trying to make. All requests like these should ultimately be informing some decision, whether that it is a big one-off decision for the company or an ongoing decision, like metrics that inform a manager on whether their team's execution is on track. When you understand the decisions driving requests, you have an opportunity to figure out how best to inform the decision beyond the specific request and how to formulate it into the format that would be most effective for the decision-makers. It also helps with prioritization since the underlying decisions can usually be sorted for importance and urgency better than specific data requests. 

The impact of doing this in my career has been that I moved from a tactical bit player with limited impact to someone who is driving strategy and has huge impact. It has also helped me coach my teams on how to have a bigger impact, which is generally a difficult thing for even pretty experienced data analysts to figure out.

Alex Justman - Vice President, Digital Resiliency at First Republic Bank:

The best lesson I got from possibly one of the worst managers I ever had “Never be victimized by your circumstances”

You have to own your situation. If things turn against you you need to figure out how to assess the position you are in, and identify who you need to work with to provide a solution to your customers!

Naomi Ceder - Chair Of The Board Of Directors for Python Software Foundation, author of the Quick Python Book, speaker, Python instructor:

35 years ago, when I was in my first job as a high school language teacher (in Latin), my first mentor urged me to learn more about computers and programming because he thought it would become important, and getting in on it in the early days would be both interesting and good for my career. He was more right than I think he could imagine since that started everything for me. 

The second advice was from my boss was to accept a posting in London to help start a new company about 10 years ago. While there were various hassles for someone older to take a position like that, for me, it was a great opportunity to expand my network and diversify my skills, which was valuable in later positions.

Matt Brattin - Founder TMB Analytics, CFO @Aplos: 

I have two pieces of advice that came to me at different times which I've never forgotten:

1) Before I got started, I asked an alumni of my graduate program something he wished he'd known before getting started, and he said to me, "There is no such thing as perfect data". It didn't mean a lot to me at the time, but definitely, as I got out into the wild, it made much more sense that the best you can do is understand your data and "respect your data" so that you can handle it with care knowing there is always going to be some nuance to it.

2) Immediately after getting promoted to my first manager role, I was told by the head of my vertical, "People are unpredictable". This was meant in a way to prepare me for people management and knowing that you can't know everything about the way people work all the time, especially when the team gets large. What I took from this, though, is the importance of communication and creating an environment for openness where you can reduce the unpredictability, but you can never eliminate it fully.

Thanks for reading, and thank you to everyone who shared their learnings ⭐.

This month, paid subscribers learned about:

• Data Portfolio Done In Notebook - How to create a data portfolio using notebooks and why notebooks are great for analytics. A walkthrough of a deep-dive analysis with data cleaning and visualizations done in Python Pandas and SQL.

• A Guide To Creating Effective Charts - An introduction to data visualization: how to select the appropriate chart for analysis and the common charting mistakes to avoid.

• Why You Shouldn’t Stop A/B Tests Early - How long should an A/B test run? It’s recommended for 2 weeks, but why? Can you stop an A/B test early? If you have to, what is the safest approach to handling fast A/B tests? What is the recommended procedure for gradually launching A/B tests over time?

  Subscribe now

Data scientists and analysts aren’t typically tasked with reporting financial metrics. Billings, MRR, EBITDA, debt-to-sales ratio, or bookings fall under the accounting and finance umbrella (thank God). We work with transactions, subscriptions, purchases, and funnel conversions, and it’s often quite a challenge to map total subscriptions to ARR or new transactions to net new revenue. It requires so much nuance and context.

People in data roles today often use ARR too freely, without understanding recurring revenue, how it differs from one-time sales, and its key components. To bridge this gap, I asked , everyone’s favorite CFO blogger and the author of the weekly newsletter on finance and strategy to introduce the ARR concept and cover must-know terms and metrics that analysts should be aware of.

An introduction to ARR

I was honored when Olga asked me to give you a lesson on the only other language I speak other than English - Annual Recurring Revenue.

As a CFO at a tech company, most of what I do translates into some impact on our topline, which we measure as ARR.

ARR is powerful because it’s essentially an annuity stream the company gets to cash in on consistently, as long as they don’t churn the customer (speaking of Churn, Olga was nice enough to write about Churn in a Mostly Metrics guest post here).

And that brings me to my first major point - why not all Revenue is equally valuable…

Not all revenue is created equal

Not to throw anyone under the bus specifically, but it’s the Wild West these days on FinTwit (that’s finance Twitter, for those not talking LTV to CAC on weekends). Pop in, and you’ll see online marketplaces calling their GMV Revenue and service-based businesses calling their one-time Revenue ARR. Your aunt’s quilt business does not fall under ARR.

So, let’s set the record straight.

1/ Revenue

This is a GAAP or accounting-based view of topline. GAAP stands for "Generally Accepted Accounting Principles", which is like the super official handbook for bean counters.

Revenue gets spread out, or accrued, to match the delivery of the product or service. In SaaS, total Revenue will usually trail total ARR and total Billings as it gets accrued over time. You’ll see in the example below.

2/ Deferred Revenue

This is the opposite of accrued revenue and largely a balance sheet and cash flow item. It accounts for money that’s prepaid for goods or services that have yet to be delivered. For example, in a 12-month SaaS contract, in month 4 there would be 8 months of deferred revenue left as a liability on the balance sheet.

3/ Remaining Performance Obligation (RPO)

RPO is all unrecognized contracted revenue. Deferred revenue goes out at most 12 months, so RPO was created to extend even further to capture all of a multi-year commitment. It includes both Deferred Revenue and any unbilled portion of a multi-year contract.

OK, let’s simplify that a bit - RPO is the future revenue customers have promised to give you, and is important for companies who sell their stuff in multi year contracts. RPO is a way to demonstrate you are de-risking the future.

For a 3-year contract, you’d have 12 months in deferred revenue and 36 months in RPO. Of the 36 months, 12 would be current RPO, and 24 months would be non-current RPO.

RPO is not a GAAP number and, therefore, does not appear on the balance sheet. Instead, companies report it in the Revenue from Contracts with Customers section of their public filings to make sure they get “credit”.

It’s really popular for consumption-based businesses where customers pre-pay or commit to lots of usage.

4/ Gross Merchandise Value (GMV)

Commonly used for marketplaces (Etsy) and payment gateways (Stripe) that charge a fee or take rate. GMV is not a true reflection of a company's revenues, but rather its through-put, as most of the revenue goes to the original seller.

5/ Annual Recurring Revenue (ARR)

ARR represents the annualized revenue run rate of all committed subscription contracts as of the measurement date. It assumes all contracts that expire during the next 12 months are renewed with existing terms.

Public service announcement:

• 1x purchases are not ARR.

• Consulting services are not ARR.

• Most ecommerce is not ARR.

Another, more nuanced, mistake is using the final year of a multi year contract, instead of the current year. Multi-year contracts with deep first-year discounting or volume ramps over time will drive deltas between the first and last year's ARR.

Many companies will claim the larger, exit year Contracted ARR (CARR) as ARR. But CARR will not track to current period GAAP revenue or billings.

Speaking of that…

If you want to trick investors, tell them about your CARR

What is CARR? It stands for Contracted Annual Recurring Revenue.

You see, in software land, you can sell multi-year contracts. And the first year may not be the same value as, say, the third year.

Many companies will claim the larger, exit year Contracted ARR (CARR) as ARR. However, CARR will not track to current period GAAP revenue or billings.

Why does this disconnect exist in the first place? There are a few reasons:

• First year discounting: You offer a customer 25% off in the first year, and then return to the base price for the out years, decreasing the revenue you actually get in year one.

• License ramp: You negotiate for the contract to increase in license count over time, with the objective of aligning to the customer’s anticipated headcount growth, hence increasing the revenue you get in the out years.

• Embedded price increases: You add in a lever for inflation that kicks in during subsequent years, increasing future contracted revenue.

The net effect of all this is it creates a perverse incentive to quote the largest annual total of the bunch. It effectively overstates the amount of business you will actually collect cash on in the current year, as well as the actual GAAP revenue you’ll record and track to.

Buuuuuuut… it makes you look better than you are at the moment! Which is why companies who fundraise often do it. It increases both the total annual recurring revenue you can tell investors about, and it artificially boosts your year-over-year growth rate. You look all dolled up for the fundraising gala.

Now, CARR can be useful in the sense that it essentially shows you the revenue you’ve de-risked down the line. But savvy accountants would argue that you can just get that from RPO (Revenue Performance Obligation) and cut the crap.

But eventually, you’ll get caught…

There will be a quarter when your investors look at your reported GAAP revenue, then look at your CARR, and then scratch their heads at the gap (not GAAP) between the two. You see, ARR should track revenue pretty closely.

And if you are growing really fast, and keep adding multi-year deals to your pile, the CARR will deviate more and more from your revenue. You become a victim of your own success in this sense as the gap widens.

And since companies are valued based on a multiple of revenue (sometimes ARR, but most deffff not CARR), investors will feel like they were duped. Not good.

And to make matters worse, the cash forecast they made during their due diligence will be off if they were using revenue and billings (which now don’t match CARR) as proxies for cash flow. So double not good.

You’ll inevitably need to come clean, or play dumb, and do a big restatement. And confronting that breach of trust is never fun. Trust me, I’ve been there.

So let’s learn how to do it the right way…

How to Build an ARR Waterfall (the right way)

Your ARR moves every day. It’s because of all the underlying dynamics of customers joining, upgrading, canceling, or downgrading. You can break down your ARR movements into:

1. ARR Gained Components

   • New Business:

     • A net new logo

   • Expansion:

     • Upgrade plan

     • Increase usage

     • Add new products

     • Raise prices (lol, my fav!)

   • Reactivation:

     • Come back from the dead, in a full, partial, or larger capacity than before

2. ARR Lost Components

   • Contraction:

     • Less licenses

     • Less usage

     • Get rid of some, but not all products

     • Price decrease (yea right!)

   • Churn:

     • Customer leaves entirely

Your “Starting” ARR is what your “Ending” ARR from the prior period. You corkscrew (a fancy word for “link”) the cells in your Excel model, and then let all the other magic we’ve listed above play out between the lines.

The ratio between New and Expansion ARR for a company varies depending on the sales model. Some companies sell a deal and don’t expand much at all. The deal is the deal.

Others will use an “edge of the wedge” or “land and expand” strategy. This is common in usage based companies where the initial deal is smaller and used as an on ramp to get customers onboarded and expanding. 

One last note - if a company has multiple products to sell, this will drive the expansion ARR up relative to the new ARR line.

Just kidding, one last, last note - you should benchmark your “churn” against other companies in similar sectors with similar business models. A monthly dollar retention rate of 99.0% equates to a monthly dollar churn rate of 1%, or 12% annually, which may be stellar in B2C creator tools but not great for B2B cybersecurity.

Thanks, CJ!

Find and connect with on LinkedIn and X.

Check some of his most popular articles: 

• How top tech companies define ARR

• Is LTV to CAC the Nickelback of SaaS Metrics?

• The top metrics for vertical SaaS companies

• How to determine which metrics you report on

• How top tech companies measure Net Dollar Retention Rate

And, subscribe to the Mostly Metrics newsletter to learn more:

Mostly metrics

A newsletter about Finance and Strategy at Startups. Written by a tech CFO. Read by world class Operators.

By CJ Gustafson

Don’t miss Data Universe 2024

Before you go, a quick reminder: the Data Universe 2024 data and analytics conference will be taking place soon, on April 10-11, in New York:

Check out the refreshed list of upcoming data conferences - The Biggest Data Science And Analytics Conferences In 2024.

Thanks for reading, everyone. Until next Wednesday!

February was a month focused on metrics, covering topics from measuring data quality to tracking product adoption. This month, paid subscribers learned about:

• How To Measure Data Quality - A consolidated list of the top metrics for measuring data governance ROI and data quality.

• How To Measure New Feature Adoption - Types of A/B tests, ways to measure new feature rollouts, how to evaluate feature discovery vs. usage, and metrics for measuring product adoption.

• Measuring Non-Cohorted Retention or Blended Churn - Cohorted vs. non-cohorted retention, the nuances of blended churn, the connection between churn and retention, and effective ways to report overall retention.

  Subscribe now

The world of data analytics is relatively small yet full of surprises. You might encounter at a meetup in San Francisco, spot while boarding an airplane, or run into Peter Fishman at a conference.

I have been following Peter Fishman for some time, so it was exciting to unexpectedly run into him at SaaStr in San Mateo last year. We discussed various topics related to data and analytics, including influencers that could be dropped into a volcano as a sacrifice. I asked Peter to do an interview for my newsletter, and he agreed! I can't wait to share it with all of you!

Peter Fishman is a data analyst, statistician, and economist. He has served as Head of Analytics, VP of Analytics, and Chief Strategy Officer and has led data and analytics teams at Microsoft, Yammer, Zenefits, Opendoor, Playdom, and Eaze. Today, he is the Co-Founder and CEO of Mozart Data, an all-in-one modern data platform for centralizing, organizing, and analyzing your data. 

Peter has seen “from the inside” how analytics evolved over the decade, and now he is directly shaping its direction. I am excited to share his perspective and thoughts on my favorite topics - how to advance in analytics, what data leadership challenges are, how to prepare for the GenAI era and secure our jobs, and more in our interview below.

What are the must-have qualities of a strong analytics leader? What does developing, growing, and maintaining strong analytics entail today?

Building data teams at technology companies used to involve identifying data talent in adjacent fields – academics, operators, consultants, backend engineers, etc. A strong data leader would lightly guide a talented team of individuals to understand the business and the key business questions. 

Strong analytics is no different today than decades ago: 

• The ability to deeply understand signals of business health. 

• Identify what areas might be sub-optimized and can be actioned on. 

• Track and analyze the relevant data. 

• Provide consumable, pragmatic recommendations.

What are common mistakes data and analytics leaders make?

Two common mistakes are over-building and antagonizing operators. 

There are plenty of reasons why, but it’s often a struggle to implement analytics wins. There’s a tension between the metric’s owner and the owner of the metrics (despite working under the same roof). 

The best leaders build efficiently, just like we’d like product leaders to do (think MVI, minimum viable insight), but also, the product isn’t just shipping the insight but capitalizing on the win by collaborating with the operator.

Recently, you mentioned something that stuck with me: over 10 years ago, we had bad tools, and most of the work was done manually. Now we have this amazing tech, yet the time to value hasn’t changed. The productivity output didn’t change. 

Why do you think this is the case?

and I have remarked on this quite a bit – our greatest impact came at a time when tools were well behind today. A couple of reasons might be that we’ve made marginal progress on the tools that matter or that the bar for valuable data insight has climbed (faster than the tools have).

I think both things are true. In baseball, it’s no longer an inefficiency to find players with good OBP (an insight from 2000, it is now obsolete), we need to comb through different data sets or take new approaches to find similar inefficiencies; the same likely holds true for the businesses that the modern data tools try to optimize.

But I think more importantly, while the tools were always important and did speed up the analyst work, the bulk of the work is to understand the business context and opportunities, and we haven’t gotten much better as a profession at that. We’ve focused on speeding up our own technical output, assuming (possibly incorrectly) that we’re already capable of the first.

Looking at the data landscape today, there are many data storage and management tools. However, it seems there isn't much focus on visualizations and analytics. Why is BI lagging behind?

The BI tools themselves are perhaps the oldest of the stack. We’ve been using Tableau (and Excel) for decades. Yes, a wave of BI and viz tools came and brought improvements to the space. There are network effects to knowing how to use a tool, which makes disruption very difficult (and costly). But there are so many flavors of BI tools (Snowflake works with hundreds of them).  

But, I don’t think “BI” (at least what we think of as BI) is where a lot of the value gets created – it’s not the visualizations, it’s the data that powers it and the impact of the folks who share it that matters.  

As a person who works on tools that make BI tools successful, it’s a bit heretical, but I think what we’ve learned is that a data team contributes most by making data present in an organization and changing the typical workflows. It’s not any specific alert or spotted trend, but rather, the mere presence of a data team gets data (and reasoned thinking) to be more considered in conversations across product and go-to-market.

There are so many exciting topics and themes within analytics. With your impressive experience and depth of understanding analytics, why Mozart data? Aren’t there enough ETL and data transformation tools? 

How about building the best analytics platform ever?

There are a lot of tools, especially ones that collectively do what Mozart Data does - we work as partners, technology consumers, or sit adjacent to many (if not all) of the tools in the space. But IT hasn’t happened. In this case, “IT” means teams universally leveraging the Modern Data Stack.  

Sure, Data Twitter and a bunch of tech companies have adopted these tools, but for the most part, it is still the case that most of the world does a lot of manual and slow data manipulation to get insight. There’s a better way, and we’ve known about it for years. And finally, the toolset is mature enough and consumable enough for a mass audience.

What is next for Mozart data? 

Mozart is still working to simplify data accessibility. This is in line with our goals to break down barriers to entry into data for technical operators and enable collaboration across technical and non-technical operational teams (pretty consistent with my thoughts above about the field). This includes improvements to our Sonata (free) plan , dbt Core integration, and Snowflake & Fivetran experience improvements. 

It wouldn’t be 2024 without mentioning we’re working on some GenAI features. Though many are trying to use AI to master going from English to SQL, we’re really taking advantage of the other direction, including automating the documentation and cataloging processes.

In this new era of GenAI, how do you see it will transform analytics, and how can analysts prepare now to secure their jobs and continue bringing value?

We’ve built automated data engineers and data analysts at Mozart Data. At least for the analysts, it’s not there – we’ve got plenty of time. But some things are certainly true – know a skill well. Low-level work (e.g., basic counts, sums, and joins) will be automated. You can’t just report the data, you need to interpret and add value through that interpretation.

The best thing to do is get reps or practice – by completing problems and noticing the nuances that arise (why does something that seems right not work, what biases exist in the interpretation of a viz), your work up-levels, and that is the only permanent job security.

You are at so many different places at the same time - conferences, podcasts, talks, and webinars. Do you still remember SQL?

I love to say – “still got it!”  Though I occasionally discover a new syntax that didn’t exist when I was learning SQL that makes something annoying much easier – though I tend to do it the inefficient way.

Do you miss the good old days with VLOOKUPs and doing 9-5 analytics? And sleeping? 

Sometimes, a 9-5 job can feel so much more energy-consuming than a job that’s clearly taking up more of the calendar. There are many applicable cliches, but I’ve found them true.

Is there anything else you want to share to encourage or inspire people to learn data?

Though data has become a profession, it’s better seen as a lens for the world. It’s a tough one to master, as it takes technical chops and real-world intelligence to make or assess a prediction. 

If you are someone who loves to make predictions and prove or disprove them – having a deep skill for assessing evidence (and thinking of that as analysis as opposed to writing SQL) is what we do as a profession. 

So much of my personal life mirrors this – when I make decisions about who to start on my fantasy football team or what route to take to get to work – it’s all data. You’ll quickly find that developing these skills applies to both work life and outside-of-work life, even if you don’t use a spreadsheet or database for that.

Thank you, Peter!

Connect with Peter: 

• X: @peterfishman

• LinkedIn: Peter Fishman

• Email: peter@mozartdata.com

• Try Mozart Sonata: https://app.mozartdata.com/signup

Subscribe now

Data and analytics are going through a massive wave of tooling that is rapidly accelerating. You will likely fall behind unless you read a hundred newsletters, watch webinars every other day, and attend data workshops. And, if you do all those things, you are likely overwhelmed.

• What is the right way to set up your reporting today? 

• Should you use semantic layers? 

• Do you need dbt? 

• Which data setup best supports a hundred metric dimensions? 

• Should you prioritize reporting flexibility or consistency?

Changing reporting setups is both costly and time-consuming, so it’s important to get it right. To help us figure it out, I invited David Krakov, who has been building data products for the last 20 years, to share his insights on how and where to build metrics. David is a second-time founder (his previous startup was acquired by Starburst, a data lake analytics platform) and the co-founder of the YC-backed and Snowflake Startup Challenge finalist Honeydew, the semantic layer native to Snowflake that helps standardize metrics, reduce their analytics engineering efforts, and eliminate duplicate logic.

In this publication, David discusses various strategies for automating metrics, ranging from ETL/ELT and BI to leveraging a semantic layer, and highlights the specific scenarios where each approach proves most effective.

Where To Build That Metric

For a user, a metric is a number they care about in their favorite dashboard or tool.

For its maintainer, a metric is something to worry about: maintain its consistency across use cases, ensure it is up to date, enable debuggability to understand a change, and allow reusability for downstream logic. A misplaced implementation can lead to duplication, data errors, and frustrated users.

Have it wrong, and your awkward conversations/month metric starts to grow.

An awkward conversation starts with a number that looks silly. “Why did it change?”, “Why didn’t it change?”, “Why does my dashboard say differently?”, “I know this is wrong”, “How did you say you counted it?”, “What did you do?”

The good awkward conversations find the culprit in something that changed in the business. The common ones find a bug in a query or some Tableau being out of sync. And in the bad ones, the messenger gets shot.

In every data stack, there are at least four different places where a metric can be built:

1. During ingestion (in ETL or a data transformation tool).

2. In the data warehouse (in ELT, using a tool like dbt).

3. In the BI tool (like Looker, Tableau, or PowerBI).

4. In a standalone semantic layer (like Metricflow, Cube, or Honeydew).

Each one is a compromise between reusability, consistency, and performance. Good compromises keep awkward conversations low, and business value high.

This post aims to provide analysts with a framework for determining the optimal method to build a business metric.

What is, technically, a metric?

For a small side quest, it is useful to define what a metric actually is. One way to skip the quest and head straight to the “Where do I build it” section is to think of a metric as “a count query that users want to run.” That’s what it is!

However, one important thing to remember is that regardless of where the metric is built, it should have every technical property. I tend to think of this as a mental checklist: You’ve got a query to run. Did you bring all the parts with you?

Aggregation Logic 

Every metric applies some aggregation to the data:

• The simplest ones are just a function call: be it your basic SUM(revenue) and COUNT(DISTINCT user_id) or a fancy approximate count.

• Filtered metrics combine a filter (segmentation of the data) - for example, the sum of revenue in the midwest or the active user count.

• Growth metrics (YoY, MoM) can combine the same metric at different times (i.e., value this month and value last month).

• Contribution metrics (per-location sales as a percentage of total sales) can combine the same metric at different levels of detail (i.e., per location and total).

Aggregation granularity

Metrics are always counted on a specific level of granularity:

• Average session length is counted at a session granularity (a row for a session).

• Revenue might be counted at payments granularity (a row for each line item).

• The first activity date might be counted at user events granularity.

Data source

Which tables are powering the metrics, and how they are joined and built:

• The per-session data might come from a single sessions table.

• Transaction data might require LEFT JOIN an order_items table with orders and customers to compute the revenue metric.

• The first activity date might be based on extracting a date key from a JSON event_user_data column in the events tables.

Time Grains

Most (but not all) metrics operate on time. How they do it is important:

• An MoM growth metric might be only computable between EOM (month-ends).

• MAU might calculated differently than DAU.

• Total sales might be available per month/quarter/year but not per day due to backfills and end-of-month processing.

Dimensions

Which dimensions can be used to slice and dice the metric by the user:

• Can an average web session length be broken down by user birth location? 

Having all data available does not mean every metric can be broken down by every dimension.

Where to build it?

Build it during ingestion

Metrics can be computed during ingestion, i.e., by updating a count whenever processing a new incoming event.

Why:

• Huge data amounts: When raw data is large and fast, computing metrics during ingestion is cheaper. If the raw data is unimportant (except for anomalies), it can even be discarded after a while. IoT-related data is a common example.

• Low latency: When a metric must always be up to date to right now, then it must be continuously updated. For example, real-time metrics such as “visitors in the last 5 minutes”.

Why not:

• Engineering process: Code that deals with ingestion is typically managed by data engineering or even R&D teams. Changes require coordination and carry an implementation cost. This makes it less suitable for dynamic business metrics or ones that analysts want to control.

• Dealing with late-arriving data and backfills: Updating an already computed metric is much harder when data arrives out of order.

• Specialized tech: Typically done with streaming data processors, in-memory databases, and other tools that are definitely not an analytics-friendly data warehouse.

• No raw data: It might be harder to connect the raw granular data that served the computation of a metric with its computed value during data ingestion to debug or deep dive.

Build it in ELT

Metrics can be computed on granular data after it lands in a data warehouse. There are different approaches to accomplishing this: one way is by building a transformation flow using a tool such as Airbyte or Fivetran; alternatively, by building queries as dbt models and utilizing an orchestrator like Airflow:

In a bare-bones data warehouse without ELT tooling, even a VIEW can be considered a type of ELT (especially once you find more VIEWs stacked on top of it).

Why:

• Consistency: A metric consumed from an aggregated VIEW or TABLE is guaranteed to be the same in every tool that uses it.

• Debuggability: The source code for the metric is an SQL query, which makes it easier to debug and understand.

• Centralized Governance: having a few places to define it (the source of a VIEW or a dbt model) makes it easier to maintain a governed source of truth.

Why not:

• Non-aggregative metrics: Every new combination of dimensions or filters requires building a new ELT query, leading to code duplication, data errors when different models compute it differently, and spaghettified code bases.

• Unpredictable downstream impact: When the metric ends in a table, it is hard to know who relies on that table afterward. Changing a definition can silently break assumptions in 500 dashboards.

• Domain experts ≠ analytic engineers: When metric logic owners rely on analytics engineers to build ELT (due to skill set or access gaps), the added communication and implementation overhead creates conflicts and friction.

• Many derived metrics: Derived metrics (for example, metrics that are built from others, i.e., contribution metrics, growth metrics, filtered metrics) complicate matters. With ELT, the initial choice of dimensions and filters is replicated for every derived metric, limiting flexibility. Different requirements for derivative metrics also result in duplication and model mess.

Build it in the consumption tool

Metrics can be defined at the final destination - the consumption tool.

This can be a data source for a BI, such as in LookML code for a Looker dashboard or in DAX for PowerBI, or Python or R in a Jupyter Notebook, or simply in a SQL query residing in a wiki page intended to be copied and consumed from a SQL IDE.

Why:

• Flexibility for the user: A metric in a BI tool is flexible for any BI operations, such as applying filters or groupings, or can be used to build derivative calculations.

• “Native” to the user: Metris in BI tools can support BI-specific implementations, including custom formatting, making them more intuitive for users.

• Non-additive metrics: A metric is computed only when a user requests it, enabling the construction of complex non-additive metrics without compromising flexibility.

Why not:

• Many consumption tools: Embedding code in one BI tool will get duplicated and reimplemented in every other tool. For example, a metric in a Tableau dashboard may also be needed for a predictive analysis in Python. Duplication across different tools almost always results in data conflicts.

• Governance: Many BI tools lack version control or a formal deployment process.

• Performance: Live queries from a BI tool can be much slower and costlier compared to querying precomputed metrics before reaching the BI tool.

Build it in a Semantic Layer

A semantic layer or a metric store is a new concept: it involves separating the layer of metric logic from both BI and ELT processes. This is to get, on the one hand, the centralization and governance benefits of ELT and the flexibility and user convenience benefits of BI.

(Bias alert: I am a founder of Honeydew, the semantic layer for Snowflake. Other semantic layers exist, such as Metricflow, Cube, and AtScale).

The way a semantic layer operates is as a tier between a user tool and the data. A user asks for a metric (”total sales by product in March”) using their tool. The semantic layer translates that request to a query that uses the standard definitions of “total sales,” “product,” and “month.”

Why:

• Centralized governance for metrics: Metrics can be treated as “first-class citizens,” making it easy to standardize their definitions and reuse them.

• No need to compromise between flexibility and consistency: Users can freely work with metrics (apply filters, slice them) in their tools. Each usage across all tools is powered by the same single definition in a semantic layer.

• Derived metrics: Semantic or metric layers excel at simplifying complexity. Have a gross profit metric that is “revenue - cost”? It's a simple expression written once.

• Distributing ownership: Semantic layers help bridge the gap between business analysts and analytics engineering.

Why not:

• More moving parts: Introducing a semantic layer adds another component to the data stack. While it shines in handling complexity, in simpler data flows, it can become just another tool to manage.

• Wild semantics: Semantic layers are typically more rigid than writing queries. They force at least some degree of design to data flows (how much varies between semantic layers). While this can promote the health of the data stack and practitioners, not every data architecture is compatible.

• Sometimes, performance: Semantic layers significantly vary in their approaches to performance optimization. Depending on the use case, they may be slower (or faster!) than traditional ELT or BI logic implementations.

In conclusion

Do not worry, it’s only a half-year project to build it again.

Thanks, David!

• Find and connect with David on LinkedIn and X.

• Learn more about Honeydew - https://honeydew.ai/product/

Thanks for reading, everyone! 

This month, paid subscribers learned about:

• Anticipating 2024: Top Trends in Analytics - How to prepare your team for 2024 with top trends and movements in analytics.

• How To Run An A/B Test On Low Traffic: What is the minimum number of users needed to launch an A/B test? How do you figure out the trade-off between confidence and test timeline? How can you increase trust in small sample tests?

• How To Set Up Subscription Analytics For Growth Reporting - How to develop analytics for subscription growth lifecycle from acquiring data to subscription KPIs reporting.

  Subscribe now

A few years ago, I interviewed for the Data Science team at DoorDash. At that time, my passion was more focused on growing analytics for B2C. Ultimately, I ended up working at MyFitnessPal. Despite this, I was very impressed with Doordash's Data Science team's efficiency, the complexity of projects, the speed of output delivery, and overall analytics maturity.

When I crossed paths with Daniel Parris, a fellow data scientist-blogger, I knew we should collaborate, and I can’t wait to share my interview with him about his time at DoorDash and his vision for analytics. 

Daniel Parris is a data scientist, analyst, and data journalist. He was one of DoorDash's first data science hires, working at the company for nearly six years, and now does consulting and career mentorship. 

Daniel is the author of Stat Significant, a weekly newsletter featuring data-centric essays about movies, music, TV, and more. He leverages data storytelling to a new level by tackling different pop culture topics through the lens of data (some of his work was featured by Fandom Entertainment).

Daniel initially majored in film and started his career in the entertainment industry before transitioning into data science. Another reason I appreciate his newsletter is that he integrates his academic interests and passion for pop culture into his professional expertise as a data journalist, creating Stat Significant. That’s what makes it so good.

Below, Daniel shares his journey as a data scientist at DoorDash, discussing his challenges, learnings, and career transition.

Data science today and tomorrow: How do you see it changing? Are there any new trends?

It's trite to say AI will change everything, but I believe AI will change fundamental components of the data science profession. On the one hand, there are the efficiency gains data scientists will reap from the automation of data coding, cleaning, and BI tasks—analysts will be able to work faster. On the other hand, there is a high likelihood that many data scientists will be responsible for building around monolithic black-box AI models. There will be a heavy emphasis on MLOps, assembling and cleaning data that flows into algorithms, and figuring out a way to interpret the outputs of these opaque systems.

The other trend I see in data science is the waning novelty of SQL and Python. Ten years ago, fluency in Pandas and scikit-learn made data scientists highly in demand. Now, there are many resources for learning these skills, or you could use ChatGPT. Technical skills no longer stand out as a unique factor; instead, critical thinking skills, demonstrating the ability to break down and solve complex problems, will become a key differentiator for analysts.

What was your most challenging project, and what were the key learnings from it?

At DoorDash, my most demanding project involved the development of the company's selection intelligence system, designed to identify the most valuable restaurants for the company to focus on.

The project necessitated the creation of a predictive model capable of estimating the value of every restaurant in the world (which was as intimidating as it sounds). These estimates would then be packaged for the sales team as leads, and they would attempt to sign the merchants onto the platform. DoorDash viewed its restaurant selection as its number one growth engine, so identifying the best merchants in every market was imperative.

The initiative consisted of three distinct parts:

1. Creating a database of every restaurant on planet earth and ensuring this repository was up to date with clean data. Every entry had to be accurate—restaurant name, address, and cuisine type—while ensuring these merchants were still in business.

2. Creating a predictive model that estimated how a restaurant would perform once it was brought on to DoorDash. 

3. Taking these predictions and distributing them to the sales team in a format consistent with the organization's compensation structure. 

Ultimately, maintaining a clean repository of every restaurant proved to be the most challenging aspect.   

Our initial merchant database came from a stale dataset of extremely low quality. Before we rolled the model out, I wrongly assumed that prediction accuracy would supersede data quality—I was sorely mistaken. The first time we distributed leads, sales reps were pissed. Nearly ten to twenty percent of lead lists were unworkable, either because these merchants were permanently closed or not restaurants at all. 

The sales team compiled a spreadsheet of the worst offenders to underscore their frustration. This Google sheet featured over 4,000 offensively bad leads, including strip clubs, consulting firms (McKinsey, Deloitte), dialysis clinics, and other garbage. I had to review the document with the head of sales to explain how this had happened. We spent the next year working on nothing but database quality. 

My major learning was to consider the end user of a model or system when launching a project. DoorDash's sales reps did not care about prediction accuracy as much as they cared about having clean, workable leads. I got so invested in building a fancy ML model that I lost sight of how the predictions would be used.

What did the data science team look like in the early days of DoorDash, and how has it changed since then?

The early days were dedicated to building infrastructure. Most projects focused on codifying metrics, creating ETL tables to report these figures, and crafting tools for business partners to view these statistics. The company was highly data-driven in its decision-making and, therefore, emphasized quantifying every aspect of the business. Much of this responsibility fell on to the data science team.  

As the company matured, the focus shifted from infrastructure-building to analysis. We had all of this data; we could track the ups and downs of the business, so the top priority became problem-solving. Analysts were called on to identify areas where the company could improve, quantify the potential impacts of strategic initiatives, and then track progress once these projects went live. There was also increased focus on experimentation, with most prominent features subject to an A/B test before launching.  

What were the most typical projects?

The most common projects were:

• Metric Definition: This work involved creating a new metric, constructing it in an ETL table, and building a dashboard to showcase results. 

• Measuring Impact: These requests typically concerned the launch of a new feature or initiative. The data science team was asked to quantify the results of these projects through experimentation or lookback analysis. 

• Identifying Improvement Areas: When there was an obvious problem with the business, an analyst was usually asked to perform a deep dive to ascertain the issue.

• Sizing Bets: During quarterly planning, the analytics team sized the impact of potential strategic bets through lightweight modeling.

Was your work stressful?

The first four years of my time at DoorDash were equal parts stressful and exhilarating. We were going up against Uber Eats and Grubhub in a highly competitive space with thin margins, so there was incredible urgency to everything we did. I was also relatively young when I started working at Doordash (23 years old) and was entrusted with the responsibility that may (or may not) have exceeded my career experience—though that was the case with most people. It was a dare-to-be-great situation where you could build crucial infrastructure for a generational company or make a multi-million dollar mistake.

After a while, the work became less stressful. The company achieved scale and had a sufficient headcount to accomplish its goals. 

If you had to start over your data journey, what would you do differently?

I majored in film in college and fell into data science a year or two after graduation. So, if I had to start over again, I'd take data science and statistics courses in college. I've suffered from imposter syndrome throughout my career because I lacked the same formal training as other data scientists. Did my atypical background hurt my job performance? Probably not. But it would have helped with confidence earlier in my career.

I still want to take night courses to expand my stats and data science know-how—to learn as much as humanly possible—but I haven't found the time. 

Why did you pivot to consultancy? Do you find it more challenging than full-time roles?

I pivoted to consultancy because I wanted more variety in my day-to-day work. Balancing clients of varying subject matter allows me to work on different parts of my brain on a given day. One day, I'll be working on a traditional product analytics task; the next day, I'll be writing a Stat Significant post; and the following day, I could be crafting a loyalty program for a fashion brand. Variety and flexibility are huge upsides to consulting. 

Finding clients is challenging, especially when you're an army of one. I've had several meetings where a potential client expresses intense interest, reaching out to arrange the call in the first place, only to have this person completely ghost me. Solo consulting means you must master marketing and sales in addition to analytics. You also have to get used to people ignoring your emails.

Your newsletter is unique and stands out from many. Clearly, you put a lot of work and love into it.

What is your motivation and ultimate goal for your newsletter? 

I love data, and I love pop culture. My goal is to spend most of my working hours contemplating movies, music, and TV while combining these passions with my data skillset. 

Every week, I get to tackle a fascinating pop culture question in a data-driven manner. Some of my favorite statistical analyses include: 

• The Rise and Fall of TV's Golden Age

• Why Do People Hate Nickelback So Much?

• Why Horror Films are Hollywood's Best Investment

From Why Horror Films are Hollywood's Best Investment: A Statistical Analysis

Stat Significant is the fullest embodiment of what's going on in my brain and is a lot of fun to write. My goal is to do the newsletter, or something derivative of the newsletter, full-time. 

How much time does it take to write a typical publication, and what is your process?

My writing process usually takes between fifteen and twenty hours per essay. For every article, I have to:

1. Find a dataset and understand the quirks and structure of the data. 

2. Perform a custom analysis on an unexplored topic. 

3. Graph the findings of my analysis using third-party visualization software. 

4. Research the topic I'm writing about in greater depth and construct a high-level essay outline. 

5. Write an eight to twelve-minute essay, providing readers with a unique data story. 

6. Edit and spot-check my work to make sure it's high-quality.   

It's an intensive process, but I believe the work is worth it (or at least I hope it is worth it). I can only manage one essay a week, but I'm optimizing for quality over quantity. My subscribers appreciate each essay's attention to detail and depth, which justifies my hard work. 

Is there anything else you want to share to encourage or inspire people to learn data?

Don't be afraid to fail. 

Christopher Payne, the former COO of DoorDash, said he learned the most when pushed outside his comfort zone and, thus, actively sought situations where he felt uncomfortable. People don't like discomfort and often associate this emotion with a threat to career stability. That said, if you never try something new, you'll never learn and grow. My failures have facilitated the most impactful learning moments of my career, and I'm now thankful for these mistakes.  

Also, failure typically leads to better stories. It's easier to connect over the things that make us human, which is our imperfections.

Thank you, Daniel!

Read some of Daniel’s work:

• Are More Celebrities Dying?

• The Fall and Rise of Nicolas Cage.

• What's the Greatest Year in Film History?

• How Has Music Changed Since the 1950s?

• Which Movies Stand the Test of Time, and Which Don't?

• What Was Watched on Netflix in 2023?

• The Hollywood Romance Age Gap.

Connect with Daniel: 

• LinkedIn: https://www.linkedin.com/in/d-parris/ 

• X: https://twitter.com/StatSigData 

• Reach out at daniel@statsignificant.com

• Get mentorship: https://mentors.to/danielparris

Thanks for reading, everyone. Until next Wednesday!

Subscribe now

I have prepared a surprise for you in this special kick-off 2024 newsletter: an interview with the famous, daring, fun-at-parties, one-and-only Lauren Balik! ⭐

It’s either love or hate with her - there’s no in-between - but you can’t ignore her bullshit-free, academically inspiring, and deciphering exciting writing. Since meeting Lauren, I’ve been fascinated by her attention to detail, thorough and in-depth research, professional storytelling, and courage.

Lauren Balik is a data leader, data wrangler, blogger, advisor, investor, and owner of Upright Analytics. She is a co-host of the Tech Bros Show with Mary McCarthy.  

Find Lauren on X and Medium.

I am honored Lauren agreed to be interviewed for my humble newsletter, and I am excited to share her always-insightful, fresh, spicy, and detailed perspective on analytics. 

Below, you will learn about:

• Navigating tensions between Finance and Data teams.

• The biggest challenges for data and analytics leadership today.

• Centralized vs. decentralized data team setups. 

• The reckoning and fate of BI tooling.

• Will next-generation spreadsheets finally take over Excel? 

• Why and how Tableau continues to dominate the market despite its challenges and limitations.

• Should you consider switching to semantic layers? 

And much more! 🔥

Today, you can become a data analyst or engineer in 3-4 months via bootcamps. Every few months, hundreds of people enter the data industry lacking a proper foundation or an appropriate degree yet can secure salaries of $130k or higher.

How does this affect the data industry?

Great question. I think this is largely on the way out.

What is often pitched by bootcamps or cloud vendors is this romanticized version of what an analyst or data engineer does. There is this mythology that a company will have its back to the wall, about to go under, then all of a sudden, this hero analyst crunches a variety of datasets and finds the one true insight to make the company millions of dollars and save the day.

In reality, most data jobs have merely become “human middleware” jobs. These are generally “glue” jobs of gluing systems together and stringing together APIs and data transfer between systems. Often times, when the glue cannot be achieved between System A and System B, the solution is to dump System A and System B data into a data lake or warehouse and then glue it together afterward.

The number of human glue jobs in the market is determined mostly by interest rates and consumer discretionary spend. When interest rates go up and consumer discretionary spend goes down, companies have less money, and it becomes expensive to have the 15th analytics engineer or 23rd data scientist on staff.

How does this affect the industry? Well, in sales, you have the concept of the ICP, or Ideal Customer Profile, the real or imagined persona of the person who will buy or otherwise transact with your offering. The proliferation of new career people and bootcamp grads has become an ICP. There are entire teams of people with titles like Developer Advocate or Data Advocate at product vendors who try to control a narrative to sell into this ICP. However, this ICP typically does not have budget authority to make big purchases, so the products are all based on install numbers, GitHub stars, and other loose metrics the vendors use to raise more money from venture capital firms.

This is why so much of the narrative is based around new tools and toys – it’s easier for product vendors to sell junk to many 25-year-olds at startups vs. selling more complete solutions to 45-year-old VPs of Engineering or Directors of Analytics at more established companies.

What do you think are the biggest challenges for data analytics leaders today?

Right now, the biggest challenge for analytics leaders is knowing where to draw boundaries. For example, in the last few years, one trend I’ve noticed and been pretty vocal about is the use of analytics teams in doing accounting. What happens is that a privately owned, venture-backed company starts growing, and the amount of SKUs or products they sell increases, and their sales deals become more complex. Now, larger, established businesses will have an ERP system, a quote-to-cash system, and purpose-built software for handling the accounting. However, those are all expensive things to buy and manage in terms of both the products and the headcount needed to manage them.

What happens is that the Finance team will often start encroaching on a capital D “Data” team or analytics team since they are already on staff, and begin taking more time and effort from the Data team to custom roll an accounting function, with logic often held together in analytics tools and on cloud data warehouses like Snowflake owning the compute for crunching the accounting numbers.

I have very rarely seen this go well. There are many issues that arise.

First, there is almost nobody in the analytics world who is skilled a Certified Public Account, or CPA. 

Second, accounting is very fluid, which is something most people outside of finance do not fully appreciate. Projections for the upcoming year, changes in accounting laws, and changes in what is considered COGS are all levers that accounting teams pull in order to present the best possible picture to investors, the market, and the government to which a company pays taxes. It’s very fluid.

When you have this fluidity going on naturally with accounting, then you add an analytics team into the mix to come up with the numbers and logic to arrive at these numbers, you have many moving parts.

I am very of the opinion that accounting should be able to close their books based on the systems they own and operate without the need for intervention from analytics teams.

Plus, I know many junior people who think they are signing up for an analytics job or data science job but inevitably get sucked into being an accountant-without-the-CPA.

It’s up to Data & Analytics leaders to push back on things like this internally.

What is your take on centralized vs decentralized data team setup? Do you have a preference? Pros/cons for each?

Decentralized, which I take to mean as analysts/data scientists/ops people sit in business units like FP&A or Marketing or Sales, is in virtually all cases the best way to get things done.

This puts the actual analysis close to the business outcomes and puts the analysis of data closest to profit centers, not cost centers, like centralized IT teams.

One of the biggest industry changes that we will see in 2024, at least in the United States, though there is a global impact, is what is called Section 174 of the IRS Code.

https://www.thomsonreuters.com/en-us/posts/tax-and-accounting/5-things-sect-174-capitalization/

I don’t think very many startups are doing this right, and I am pretty doubtful most larger enterprises are thinking about this, either.

Essentially, the amended IRC Sec. 174 eliminates the ability for businesses to deduct their R&D costs as an expense. Instead, they must capitalize these expenses and amortize them over a period of 5 years for US companies or 15 years for foreign corporations. In short, technical staff and engineers are becoming a lot more expensive.

Many data engineers and even analytics and data science (research) jobs will be impacted, in addition to product management and product engineering.

Whether you run a business unit, a P&L, a whole team, or an individual contributor, you should be asking internally where your salary rolls up. Are you R&D? G&A? or S&M? There is going to be a lot of shuffling around and likely layoffs, given that US businesses, and especially smaller businesses and startups, can no longer take R&D expenses in the same tax year. In general, this benefits big fish like Google, Microsoft, Amazon, and others at the expense of startups.

I think this will all lead to more decentralization in many small and mid-size businesses. Large headcount teams will just become too expensive to be considered R&D.

The current market for BI tools is limited. Although there are many cloud analytics dashboard tools, not many can overtake Tableau in its reach and functionality. Why do you think this is the case? 

BI tools have a relatively low barrier to entry in the current market. Because of this, pricing is a race to the bottom, especially for cloud-first and cloud-native solutions. If every vendor is cutting on price, it is very difficult for these companies to generate meaningful revenue in the mid-tenths or hundreds of millions of dollars in annual recurring revenue (ARR) to generate the returns venture capital needs.

Because of this, many of these BI tools become involved in two separate but distinct games to make them more attractive to potential buyers and thus raise their valuations.

First, many BI tools engage in a game with cloud service providers (AWS, Azure, Google Cloud) and resellers of CSPs like Snowflake and Databricks, in which the BI tools drive incremental consumption revenue back to the CSPs and CSP resellers. Many buyers should be aware of this game before and during a POC or purchasing process.

Some BI tools will make overly complex queries or suggest patterns that increase Snowflake or similar bills an extra 10-20% per year just by using them vs. earlier stage tools like Tableau. Snowflake loves it. The BI tools love it. The customer may even love the experience at first until the costs start kicking in.

Ultimately, it’s up to customers to realize that these games occur. Removing things like overly complicated and overly scheduled materializations can go a long way to saving costs and latency. Removing 5 layers of SQL or dbt SQL before data even hits a BI tool is important as well.

Tableau is so widespread because they won the battle of the User Group and of community, and it’s not even close.

In the mid-to-late 2010s, Tableau had a very low barrier to entry with their Tableau Public offering. There were many programs like Viz of the Day and similar that allowed people to submit data projects and visualizations and grow their careers and presence in the market.

No other company has allowed individual developers to become overnight data superheroes. Thoughtspot doesn’t have this. Mode doesn’t have this. Chartio didn’t have this. Even dbt Labs, which uses the idea of “community” heavily, doesn’t have this.

Alteryx and PowerBI also have similar programs that allow developers to be showcased and have their projects shared with the masses. Alteryx went public and has recently announced a large buyout to take it private again. It’s very well distributed. PowerBI is very well distributed.

The common thread here in GTM is that the most successful visual or front-end/customer-facing products with the largest enterprise values all enabled individual developers to become superheroes. These developers then show off their work on social media, they get job opportunities, and they become known in data and analytics pockets as experts.

Could Looker do it?

In 2018/19, I remember being fascinated by Looker, its version control, and embeddings. It felt like the tool of tomorrow. LookML was easy to learn and intuitive. Eventually, it disappointed me with the lack of custom visualizations and charting functions.

And yet, as you know, I’d still choose it today over static Tableau workbooks.

Looker did not necessarily follow the same playbook in their lifecycle. In the early days, a lot of Looker’s growth was fueled by their great professional services and customer success teams. Businesses were basically hiring “SQL experts,” and Looker was the product wrapped around it. Looker had a great semantic layer and was a sticky product because once an organization commits to a semantic layer, they are going to continue with that vendor. All the intermediary SQL is secondary.

Is using semantic layers the path to take?

I’m not sure what to make of the latest wave of semantic layers. I am not sure there is much benefit a semantic layer offers vs. a traditional OLAP cube.

The semantic layer sitting on cloud SQL, on consumption-based data warehouses, is mostly just a way to drive more incremental consumption.

What I think is much more interesting is the work being done outside of the SQL-only world. For example, I recently met with the founder of Brim Data, Steve McCanne, who showed me his view of the world in which data types are a first-class citizen. Right now, all the Avro, Parquet, throw-it-into-warehouse, SQL compute-it-out way of thinking about the world is highly inefficient. What if data types were a first-class citizen? This would remove a lot of the reformatting needed in managing data. Right now, data is being transformed down and rolled back up, passing through 4-5 people at a minimum, and it’s all just a factory line.

I also think another real battle is in change data capture. Right now, the “Modern Data Stack” is merely just a tax on latency. If you want better latency, the easiest way is to crank up the computing resources involved, which very quickly becomes expensive. Things like managing data schemas before dumping the data into location number two is way more efficient than relying on the raw compute resources of location number two to write layers of business logic.

Are dashboards really dead? [refereing to Thoughtspot ebook]

Dashboards are not dead.

The word dashboard in data is literally stolen from a car and airplane dashboard. A dashboard tells you how to get from Point A to Point B. How fast are you going, how much gas is left in the tank, and are there notifications that need to be addressed, like a check engine light?

But the important thing is that you can’t take actions on the car dashboard. You can watch your fuel go from “Full” to “Empty” on the dashboard, but you can’t actually fill your tank with the dashboard.

I am very confident that every dashboard should have a line graph showing a trend over time, a big number showing the KPI for the day/week/quarter/year, and then Top N. If you want to go a step further and add a full table, go ahead. If you want to add a color or notification to show anomalous behavior, go ahead.

One very funny thing is that ThoughtSpot recently bought Mode, and Mode is heavily into the idea of the dashboard. I’ve seen companies where there are more Mode dashboards than employees working at the company by a factor of 2 or 3. So how can ThoughtSpot say the dashboard is dead if they just paid about $200M for a company that is based around the concept of dashboards?

Dashboards will continue to be both dead and undead at the same time, like Michael Jackson in the “Thriller” music video.

Over the last few years, there has been a wave of “next-generation” Excel and spreadsheet tools (e.g., Equals, Quip, Arcwise, Zoho, etc.) 

Do you believe they could finally take over Excel?

I do not, and the reason why relates to metadata and collection and privacy reasons. Maybe this can be considered a hot take, but it is something that I know prevents distribution at many companies.

Many of these Excel alternatives are simply Excel-in-the-cloud. They collect a lot of data on users, which can include browsing patterns and metadata about what is specifically being run through the platform. 

For example, if you have a table called “all_orders,” which means all gross orders your business does, and each row in the table is a gross order, you are passing through usage data about the health of your business (how strong your sales are) to many of these vendors. In the Terms and Privacy Policies of all these vendors, you can see that most of them reserve the right to sell or use your data to “improve the services,” which legally means nothing.

If, for example, your business traditionally does 1000 orders a day on average, then all of a sudden you are only doing 500 a day on average over a few weeks, you are passing this data to these data vendors that help you make your reports. They will be able to detect these patterns and then use this information about your business if they wish.

This is a large reason why many of these cloud-first platforms struggle with enterprise. Many enterprise customers simply do not want to give away usage patterns about their business health.

The upside and benefits of these spreadsheet platforms are that they may offer some niceties over Excel, but in my experience, Excel, which can be run locally, is too widely distributed and too popular to be overtaken.

You dedicate a lot of time to researching different data platforms. What tools, companies, or technologies do you appreciate and recommend?

This is an incredibly silly answer, but I promise it goes somewhere. I have something called “Mascot Theory” that has become popular in investment circles. The idea is that the more a business uses mascots to sell and promote their offerings, the worse their offerings are, and that once mascots start appearing, there are unfavorable terms and hidden costs the customer may not understand when signing up.

Take, for example, Payday Loan or Cash-For-Gold places that you may see across the US, in small towns, in rural areas, and even in big cities. Many of these places have sign spinners out front or even an Uncle Sam on stilts. Most people know, of course, that they aren’t getting good rates or offers from these places, but because these businesses use mascots, it takes away the pain for the customer of knowing you are getting the bad end of the bargain.

It’s the same with banks. If you go to a baseball game and they have Free Bobblehead Night or Free Bat Night, where all the kids get a free bat, these are almost always sponsored by banks and almost always have the sports team’s mascot shooting out T-shirts out of a cannon with the bank’s name on it.

Why is this? Well, when you add a fun mascot in the mix, you forget that that bank owns 50% of the equity in your house. You forget that the bank declined you for a credit line increase on your credit card. You forget that the bank credit card has worse interest rates than other credit cards you could be using.

It’s all based on mascots. If thousands of people at the ballgame are laughing and everyone else is playing along with the mascots, and we all collectively look at each other and agree we are all having fun, we forget that the bank that’s putting on the show owns 50% of our houses and charges us high-interest rates on credit but low returns on our deposits.

How does this relate to the data world? It’s very simple.

Snowflake, for example, has a well-known polar bear mascot. You can hug the Snowflake mascot or take a picture with it at all kinds of events. Snowflake, too, now has superhero mascots floating around in costumes at various events. Some of these are hired actors, and others are various analytics people and engineers they put in capes. So now we’ve got a polar bear mascot, plus superhero mascots in the Snowflake ecosystem. 

Salesforce has about a dozen mascots. If you go to a Salesforce event, there are many costumed mascots floating around. In fact, some in the Salesforce community even dress up as their favorite mascots and post these pictures on Instagram and other social media!

Why do these platforms have dozens of mascots? Well, it is because these platforms have the worst unit costs. The more silliness that goes on, the worse the rates tend to be, just like with the bank, just like with the Cash-For-Gold place.

Google Cloud Platform doesn’t have mascots. Databricks doesn’t have mascots. For example, there is no Bricky the Databricks Brickster dressed up like a Databricks brick. AWS has started in on mascots lately, and they now have an S3 data storage mascot.

It’s true!

Of course, you can blow up spend and make all kinds of poor solutions on any platform – there is no shortage of writing bad code or designing inefficient solutions, and every platform vendor is happy to take your money. However, I always lean toward staying away from mascots.

I know this answer will get me in trouble with the Snowflake people so I’ll say that Snowflake can be a great solution to get started, there is no denying that.

How about data influencers and experts - whom you enjoy reading and keeping an eye on?

There are no good data influencers. Data influencers can be dropped into a volcano as a sacrifice.

How do you see this new era of GenAI transforming analytics?

I believe most text-to-SQL is going nowhere, and those companies are a dime a dozen and will put out or pivot entirely in the next year. I am not especially bullish on anything that seeks to improve BI or improve tooling.

What I am interested in is the ability of AI to improve audiences and consumer privacy. Right now, the entire analytics world is set up to:

• Collect click and event-based behaviors

• Collect purchase behaviors

• Throw this all into models and recommendation engines and return back to the consumer what a company *thinks* is what the consumer wants, which is then delivered through ads and promotions.

But all of this is wrong, in my opinion. Companies already know who their top 1%, 2%, 5% spenders are, and these customers are always an outsized portion of revenue.

Instead of companies making best guesses about what these consumers want, both parties should have tighter feedback loops, and these consumers would ideally want to give more data to these brands they like and trust and give less data to brands they don’t like or trust.

Right now, ad networks have the most power, brands have the second-most power, and consumers have the least power. Consumers have their data sold 10 different ways across brokers, networks, it ends up in hedge funds used to price equities, it’s all a mess.

The cookie is going away after years of Google playing “will they or won’t they” with the market. Gmail is also making it harder for bulk senders of emails to spam customers in 2024.

All of this means that it is now more expensive to acquire new customers, and these dollars can probably be better allocated to retaining top existing customers. This means more personalization and more experiences. AI will be best deployed in these retention use cases.

Is there anything else you want to share to encourage or inspire people to learn data and analytics?

All of analytics and data engineering is just applied accounting. The more you know about finance, the stronger you are as an analytics professional.

Thank you, Lauren! 

Find Lauren on X and Medium.

Check some of Lauren’s popular publications:

• The Modern Data Stack Through ‘The Gervais Principle’

• Introducing the Tech Bros Show: A Higher Interest Rates Podcast ⚡

• Open Letter to the Board of Directors of GitLab Inc. (NASDAQ:GTLB)

• Customer Empathy is Dead

• Against Config as a Lifestyle

• Don’t Do Accounting in dbt Cloud

• dbt Cloud’s New Pricing Model: The Sinister Phase II

• 6 Reality-Based Predictions for Data in 2023

Thanks for reading, everyone!

This month, paid subscribers learned about:

• Applying Statistics In Product Analytics - A deep dive into distributions, their types, use cases, and examples to help you decide which distribution to apply for your analysis or forecast.

• SQL For Weekday Product Usage Analysis - How to get the frequency of user engagement using SQL for day-of-the-week analysis to understand cyclicality and usage patterns.

• When Things Go South - Or why self-service is a myth, why it may introduce discrepancies and inconsistencies in reporting, and how it can damage your data culture.

  Subscribe now

I have been following and reading for a long time, even before my iCloud storage maxed out with screenshots taken of his LinkedIn posts. I have adopted many of Ben’s data handling and processing concepts from his publications, videos, and tutorials.

There are now many data engineering influencers and experts. Still, to me, Ben stands out from most as he brings theory into real-life scenarios with a detailed, unbiased, and thorough approach to problem-solving. I also enjoy how structured and easy-to-read his content is and how it’s tailored to all levels of analysts and engineers. His Substack is on my list of top newsletters about data and analytics.

While I disagree with Ben on a few concepts (come on, analytics is way more impactful, challenging, and interesting than data engineering. And it’s the best possible career growth!), I have long been a fan of his newsletter and YouTube channel (Seattle Data Guy). 

This week Ben is celebrating his 100th newsletter 🎉🎉🎉, and I’m so excited to interview him for my journal! 🤩

Ben Rogojan is a data infrastructure and engineering consultant. He helps his clients set up and improve their end-to-end data processes and workflows as the Seattle Data Guy. This could be helping clients migrate to the cloud, improve old infrastructure, or help create data sets for ML models. Before consulting full-time, he worked at Facebook, a healthcare analytics start-up and a large healthcare provider. He also loves sharing his thoughts on his two newsletters (one on data infrastructure, the other on starting a data analytics consulting company) and YouTube. 

For analysts transitioning into data engineering, what skills and qualifications do you think are a must today?

There are a few key skills I’d recommend an analyst pick up if they want to transition into data engineering. 

To start out with, you’ll likely want to improve your SQL (or at least think about SQL from a more structural perspective vs. purely an ad-hoc one) and learn to program; Python is fine unless you plan to work somewhere that uses Scala or Java and then from there the key is understanding data warehouse and data pipelines. That’s usually enough to get you the job. From there, you’ll pick up so many other skills!

What’s your take on analytics engineering? Do you see it as a transitional role tied to specific tooling/strategy and isn’t easily plugged in at any company or, rather, potentially its own domain that will go into its further specializations soon?

I find that analytics engineering captured the fact that data engineers often fall on a spectrum in many companies and teams. Those heavily technical in spinning up Docker containers prefer Scala over Python. On the other side, some data engineers tend to focus more on data modeling, Python, SQL, and how to make the data easy for analysts to use. The former is where I find analytics engineering has stemmed from. 

In terms of its longevity, I think that function will always be required, agnostic of tooling or title.

You pivoted from a full-time data engineering role into a consultancy. I did the opposite - after 5 years in consulting, I transitioned back to full-time. I miss the opportunity to work with different tech stacks, but I found consulting rather challenging overall. Any regrets?

I don’t have any regrets. 

Transitioning from a full-time data engineer to a consultant has opened up many opportunities for me. Both in terms of learning opportunities (like getting better at selling projects and ideas) that have helped round out my skill set to getting involved with projects in various industries. 

Overall, it’s been a fun journey with a whole new set of challenges:

Dcubed’s Substack

My First 6-Figure Consulting Project

By Benjamin Rogojan - AKA (The Seattle Data Guy) After signing my first 6-figure consulting contact, you’d think I’d be excited. I was. But honestly, the most palatable feeling was not excitement…it was an impending dread. I had inked my first signing contract. Up until this point, I had been taking on 10-20-30k projects. So why was I feeling like I had sud…

Read more

3 years ago · 8 likes · Dcubed Consulting

What was the most challenging project you ever had to work on? 

I think one of the more challenging, only in terms of logistics, was a project where I had to work in a heavily regulated industry. Actually, since I didn’t have the proper credentials to work in said industry, I had to essentially write code and Airflow DAGs on my computer and have someone else deploy it on the actual system it’d be tested in. It was like playing a game of operator sometimes. Having to tell a person how to enter Vim, run a script, or start up a docker container (they were a Windows person). It really was more of a test of patience for everyone involved.

Every year learning data resources become more accessible, affordable, and better. Today data enthusiasts can learn new, highly demanding skills for free. This wasn’t the case for us years back when we entered the industry. 

Do you feel the quality of new data people changes as well? 

It’s honestly amazing seeing all the new resources available to learn about data, software engineering, and just about every other technical skill set. I think the potential to accelerate your career is far better than it ever has been. Even about a decade ago, when I started in the data world, you’d have to put together your own courses on data engineering essentially by finding a Java intro from the NewBoston and a SQL course from WiseOwl while picking up a book here or there. 

But now you can find fully written guides, and you have data engineering projects from great creators like Darshil, and BigQuery and Snowflake both offer free data sets (back when I started, you had to spin up a SQL Server or Postgres instance and find some CSV to load into it).

The only challenge now is keeping motivation. I have written about it in the past, the curse of tutorial hell:

“We have all fallen into tutorial hell. More than likely, for most of us, it all started with that first Python tutorial or advanced SQL course.

To some degrees, it is part of the learning process; the constant repetition of the same basics over and over again until something sticks. But to really allow the new subject a permanent residence in our minds, we must find methods such as accountability groups and projects to ensure we don’t lose what we just learned.”

So I imagine at this stage, we probably have more people becoming analysts and data engineers because the barrier to accessing training, tutorials, and information has lowered. I would say I have noticed what feels like a broader set of individuals who write SQL and a little Python. I can’t truly speak on quality, as it’d purely be anecdotal.

What are common mistakes you notice analysts (or data engineers) often make?

• Attacking the technical problem without understanding the business problem - One of the common issues I see a lot of junior and senior engineers/analysts is jumping on the technical problem vs the business problem. Now generally, if you’re a senior level, you probably do this less often, but I do think it’s an easy trap to fall into. Perhaps you just learned a new skill or have some recency bias in terms of how to solve a problem. Instead of pausing and asking the business what they actually need, you may just jump into building which is a gamble in terms of whether or not you will deliver what is expected. 

• Building systems that are hard to maintain - It’s really easy today, with all the various tools and technologies, to want to use everything even if they don’t actually solve the problem. It doesn’t help that job descriptions seem to push this narrative that you need to have 30 different technologies to be hired. 

• Not having some form of data analytics process - One thing I learned early on is that if you don’t apply some form of analytical process, you are likely to create an incoherent end product. Either the product didn’t go deep enough, failed to answer the real question, or never got a conclusive end.  

What would you do differently if you had to start your career over?

I think the one thing I might have done differently is to spend even more time on the fundamentals. 

The tech world is one where you constantly have to keep up with changes, at least in terms of understanding how they impact your daily workflows. And honestly, the more firm a foundation you have, the easier I believe you can pick up new technologies. 

Spot on. While interviewing, I ask candidates their SQL level from 1 to 10, with 10 being the most advanced. Interestingly, 6-7 years ago, most candidates put themselves at 5. Now, many candidates rate themselves at around 8-9 and yet still fail to explain window functions or SQL cost optimization practices.

There seems to be a constant desire to rush through the learning process. There are plenty of 12-week bootcamps and accelerator courses to learn new technology. But our careers are often pretty long, and I don’t think there should be a constant rush to speed through learning. 

If you can, take your time and enjoy learning. Because at a certain point, you’ll be expected to learn new technologies, manage projects, get buy-in for next year's budget, and grow a team. All of which will become challenging if you don’t already have a solid base in fundamental data, programming, and analytics concepts. 

Is there anything else you want to share to encourage or inspire people to learn data?

We are entering an exciting time in the next 5-10 years. Many people ask me if I think data engineering will be around in 10 years, and that’s too long to predict anything. I think there will always be a need to manage data, and the data we are being expected to collect and manage is growing in all senses of the 5Vs of big data. So even if we have LLMs and some generative AI, I just see them as tools (at least for the next few years) to assist us.

After that, who knows?

Thank you, Ben! 

Check some of Ben’s most popular articles:

• My First 6-Figure Consulting Project

• The State Of Data Engineering

• A Zero ETL Future

• Getting Unstuck In Your Data Career

• How And Why We Need To Implement Data Quality Now!

• Building A Million Dollar Data Analytics Service

• Centralized vs Decentralized vs Federated Data Teams

• Becoming A Better Data Engineer - Tips On Translating Business Requirements

• Is Everyone's Data Infrastructure A Mess? (the answer is yes, but no). 

And videos from the Seattle Data Guy channel:

• Data Engineering Road Map - How To Learn Data Engineering Quickly

• Top Courses To Become A Data Engineer In 2022

• Intro To Databricks - What Is Databricks

• What Is DBT and Why Is It So Popular - Intro To Data Infrastructure

• Being A Data Engineer: Expectations vs Reality

• How To Become A Data Engineer in 2023 - From Coding To The Cloud

• The Future Of Data Engineering | Where Data Engineering Is Going in 2023

Make sure to subscribe to Ben’s newsletter to learn more:

SeattleDataGuy’s Newsletter

Learn About End-To-End Data Flows (Data Engineering, MLOps, and Data Science)

X: https://twitter.com/SeattleDataGuy

LinkedIn: https://www.linkedin.com/in/benjaminrogojan/

Thanks for reading, everyone. Until next Wednesday!

This month, paid subscribers learned about:

• SQL And Python Mistakes To Avoid - where I grudgingly share some painful lessons I learned from my own errors made while working with SQL and Python. 

• A Deep Dive into Onboarding Flow Redesign Analysis - I put together everything I learned so far about re-designing and analyzing user onboarding into a guide with benchmarks, process, metrics, testing paywalls, and the expected unexpected.

• Statistics 101: When Simple Becomes Tricky - There is no excuse for failing to replicate basic functions during your data science interview. I break down variance, standard deviation, mode, median, and distribution to prepare you for what you will be asked to explain.

  Subscribe now

As you probably know, I am not a fan of meetings. I believe standups, check-ins, weekly KPIs readouts, and project status updates are redundant. They waste everyone's time and could be replaced with smart, efficient, and automated reporting.

I was exploring automated ways to share weekly KPI reports and ran into Push.ai, after which I immediately fell in love with their report builder. I liked their metrics monitoring, visuals, and simplicity. So I reached out to them and met with Britton Stamper, analyst, data storyteller, CTO, and co-founder of Push.ai. After a fun convo about semantic layers, Periscope (I used to be a fan), and learning more about push.ai, I asked Britton to share their approach and industry best practices to report automation. He agreed to write for me, and today I am so excited to present my third guest writer who walks us through the process, tooling, and steps on how to automate and streamline data reporting.

How to accelerate your data reporting with automation by Britton Stamper

Reporting is one of the core deliverables for data analysts. However, the most common solution of manually creating reports can be time-consuming and leads to inconsistencies. Data analysts often get caught in a trap of updating the same set of reports every month, copying data from source systems into documents, spreadsheets, or slides. Even if manual reporting has been optimized to take as little time as possible, there is a high chance it can introduce errors and discrepancies.

My first job out of college had 3 distinct sets of eyes checking all the analytics and reporting in case someone had mistyped a number of copy-pasted poorly from the source system. Eventually, manual reporting becomes a bottleneck as data analysts want to increase the impact they bring to their organizations.

In this publication, I will share how to get started with automating reporting and setting up the systems so that data updates and distribution can be automated. By the end of this, you should be able to see your own path to create consistent, automated, effective reporting.

Getting started - define measurements, dimensions, and metrics

The first step in automating reports is to align with key stakeholders on what data they need and how they would like it transformed from the source systems into metrics they want reporting on. Data used in reporting fits into one of three categories:

1. Measurements: Important aggregations of data that business teams care about (e.g. Order Count or Revenue)

When defining measurements, dimensions, and metrics, strive for clarity and consistency. The key assumptions, data sources and update frequency (SLA) should be written out so that everyone agrees on the data in reports. This process can be written out as detailed descriptions in a shared document. This keeps stakeholders involved and aligned on the outcomes of a report automation project.

2. Dimensions: Data used to group measures into categorical information (e.g. Customer Segment or Product SKU)

One particularly important type of dimension is the time dimension. Reports are usually time-based, set up on some cadence such as weekly or monthly, to allow teams to analyze changes in measurements over time. Combining a measurement with a specific time dimension allows teams to define metrics.

3. Metrics: A measurement over time, often used to track progress (e.g. Order Count by Created Date or Customer Count by First Payment Date)

This may sound simple, but it is the most commonly overlooked step in creating automated reporting. First, teams need to establish when a particular measurement is significant for the business. If we take an e-commerce business, is the revenue recognized when an order is created, when it is shipped, when it is delivered, or when the return policy has passed and the revenue cannot be reversed? There is no universal standard yet for metric definitions so teams need to make the measurement and time dimension relationships clear by documenting the metrics that matter to their organization.

One additional place where the teams we work with struggle is getting the time dimensions to align for reporting. The most common type of time dimension confusion comes with conversion metrics. If a team is reporting on the conversion for an activation rate, such as a 7-day activation rate where the metric is only valid after 7 days from the user’s created date, then the time dimension must be translated to align with the conversion rate definition. This can be done by using a date_add() or + interval ‘7 day’ function in SQL:

SELECT 
  (created_at::DATE + INTERVAL '7 days') as shifted_cohort_date,
  COUNT(user_id) AS cohort_size,
-- Using conditional aggregation to count users who converted within 7 days
  COUNT(CASE WHEN DATE_PART('day', converted_at - created_at) < 7 THEN 1 END) AS converted_users,
  (COUNT(CASE WHEN DATE_PART('day', converted_at - created_at) < 7 THEN 1 END)::FLOAT / NULLIF(COUNT(user_id)::FLOAT), 0) AS seven_day_conversion_rate
FROM Users 
GROUP BY (created_at::DATE + INTERVAL '7 days')
ORDER BY (created_at::DATE + INTERVAL '7 days')

Common Report Calculations

Report calculations can be simplified to some basic queries that perform time aggregations. These queries either GROUP BY time dimension or use the time dimension in a filter like WHERE date BETWEEN start AND end. By aggregating by time or comparing similar periods, teams can understand how metrics are changing and make more informed decisions. 

Common calculation types are:

• Intraday:

  • Hourly: Useful for operations that need near real-time insights, such as monitoring web traffic or e-commerce operations.

  • Minute-level: Useful in scenarios like high-frequency trading or ad bidding where every minute matters.

• Periodic (Daily, Weekly, Monthly, Quarterly, Yearly):

  • Prior Period: The last complete period's data.

  • Period Over Period: Compares data from one week to the previous week.

  • Period to Date: Cumulative data from the start of the current period up to the current date.

• Rolling or Trailing Periods:

  • Trailing 7 days: The past 7 days, excluding today.

  • Trailing 30 days: The past 30 days, excluding today.

  • Rolling 12 months: The past 12 months, including the current month.

• Custom Periods: Depending on specific business needs, data may be aggregated for any custom time period like the last 10 days, last 6 months, etc.

• Seasonal: Comparisons made for specific times in a year, e.g., Christmas season, back-to-school, or summer months. This helps in understanding seasonal trends and patterns.

• Event-based: Aggregating data around specific events, like a marketing campaign, a sales promotion, or major global events like the Olympics.

Time analysis is easy to perform in SQL. In the first query below, we pull the periods over time by truncating the date into a useful calendar period like month or week. In the second query, we create a custom set of date expressions that can be unioned together, where the start and end dates create the two comparable periods.

Creating periodic time series data:

select date_trunc(date_field, ‘week’) as period, 
  {aggregation}
from …
group by 1
order by 1 desc

Creating period comparisons:

select {metric}
from …
where date_field between recent_start_date and recent_end_date

union

select {metric}
from …
where date_field between prior_start_date and prior_end_date

The first query allows teams to perform time series analysis, usually displayed in a chart or table. If metrics have been defined clearly, the charts or tables should all line up. For reports that show many metrics, it's best to either display them in a single table or a series of charts, where no chart has more than one or two metrics. 

The second query allows for a wider range of comparisons, such as current week-to-date versus prior week-to-date, and can be used to tailor reports to individual report needs. A team can use the week-to-date comparison to understand the pacing of their current week’s performance against the prior week’s.

Agree on the Report Wireframe and Formatting

As teams start to automate reporting, they need to start implementing a consistent structure. Data teams can save time and ensure a standard practice when creating new reports. Templates reinforce strong data governance, where calculations have consistent definitions and usage. Finally, the templates also provide stakeholders with a consistent way to see data so they can read over larger amounts of information quickly and accurately.

Here are a couple of wireframes of report layouts that Push.ai uses to show metrics and their period-over-period change:

When structuring reports, make sure: 

• There is a balancing of data values and visuals. While charts offer visual insights, they must complement, not overshadow, the data. 

• Put the most important information at the top of the report to establish context. 

• Provide supplemental information in hyperlinks so that it does not distract the reader. These links keep the core report focused while providing more complexity that teams may need to investigate further. Often, teams link to dashboards, charts, or documents that provide more context and analysis.

• When adding more data to reports, be critical of all the elements already in the report. Either add the new data into a relevant section or deprecate a section that is no longer needed. 

• Finally, avoid information overload. Data analysts commonly fall into the trap of adding new requests to the bottom of an existing report. This can make reports less readable and cause confusion. Reports should convey key insights without drowning the reader in a sea of data.

Here’s an example of a basic report that could be sent out in Slack:

In this example, in the following report, the numbers are the focal point to keep the team focused on the changes day over day. This could be complemented with a link to a dashboard to drill into data points such as why orders were down 4% yesterday.

Now that the metrics and reports have consistent definitions and structure, they can be automated.

Automating Reporting

Step 1 - Choose a data stack

To get started with automation, data teams need to invest in infrastructure to produce the data needed for reports. Teams often rely on a modern data stack for low-cost, easy-to-implement automation. There are many articles on the modern data stack and the categories that provide value. A general modern data stack includes:

• ETL (Extract, Transform, and Load): pull data from the sources into a single database

• Data Warehouse: efficient storage and computing database designed for analytics

• Data Modeling: transform data in the warehouse into useful reporting structures

• Semantic Layer: a single definition layer mapping data to business concepts such as metrics and dimensions.

• Reporting Layer: tools that provide a data interface from the warehouse to the business teams, often as dashboards, charts, and tables.

Note: Before investing a large amount of time and resources into report automation, ensure that you have picked a high enough value project that will justify the investment in automation.

Step 2 - set a reporting cadence

The value of reports lies in the timeliness of metric updates and insights. For example, if strategic sessions are on the first day of each month, then reports should be ready before the monthly meeting. More frequent reporting cadences such as weekly or daily can be helpful to keep teams informed of changes and take action faster. Aligning the reporting cadence to each report’s use case makes them much more impactful. A good proactive step is to set a reminder to check reporting cadence every 3-6 months so they can be adjusted when needed.

SLAs (Service Level Agreements) are a concept from support teams that establish a minimum time to perform an action. An SLA describes how quickly data will be refreshed for reporting, such as within 3 hours of latency. Having data SLAs allows teams to establish the maximum frequency that reports can be set to. 

For the data team, faster refreshing can cost more money due to increased computing and number of jobs that need to be run. The data team should try to balance these costs against the requirements of the business. Business teams usually don’t understand this trade-off of cost versus latency. The data team and business team can align on what level of cost is worth the increased frequency of updates. There will be a point where the business team receives less value from higher frequency and the data team incurs a higher cost that is not worth the tradeoff.

Step 3 - agree on the distribution

It’s important to make sure that stakeholders receive reports in tools they are comfortable with. Most tools have some form of automation available, such as scheduling dashboards to be sent on a regular basis from BI tools like Tableau or Looker or having data connected to Google Sheets. Make sure that reports are sent where stakeholders will see and use them. 

Step 4 - run data quality checks

There are a lot of emerging technologies for data observability, such as Metaplane and Monte Carlo, that will test data for large, locate unexpected changes and then send an alert to the data team. You can also run additional custom tests using dbt or similar tools to ensure metric values aren’t changing. These tests allow data analysts to make sure that the data reporting is providing consistent results.

For an advanced implementation of custom data quality tests and governance, you can read the Gitlab Handbook page on data quality.

Step 5 - set alarms and escalation paths

The last step with automated data reporting is to have a clear escalation process whenever a data quality issue happens. Errors and outages will happen at some point and being prepared with a pre-agreed process will allow it to be much less stressful and have less of a negative impact on the business. 

Below is a list of steps in a common process. Not all the steps will be required for data quality issues, but most can be helpful, especially in communication.

1. Clearly communicate the scope of the outage to impacted stakeholders.

2. Assess the situation and map a course of action.

3. Communicate updates as fixes are put in place.

4. Send an update when you have a solution in place and have begun monitoring.

5. Send an update when the solution has been confirmed and the outage is resolved.

6. Host an internal data team post-mortem to prevent this type of outage in the future.

The Future of Data Reporting

Automating reporting is not a new concept. It has been around since modern computing allowed people to offload calculations onto machines. However, there has been plenty of innovation in reporting, often subtly such as the ability to connect Bigquery directly to Google Sheets. Teams are continuing to tie the source of truth for data, the data warehouse, to the tools they use to interact with data such as spreadsheets and BI tools. 

At Push.ai, we see the future of reporting becoming possible with a focus on creating consistent metric and dimension definitions. We’ve built a way for teams to write metric and dimension definitions directly from their data warehouse so that they can quickly set up any type of reporting they need. Even better, with emerging open-source semantic layers like dbt or Cube and closed-source modeling like Looker’s LookML allowing teams to write definitions in code, the metrics and dimensions can be reused across any data tool. With data definitions centralized, anyone in an organization can find, subscribe and receive the exact data they need, when they need it. We make it easy to send data to the places where teams collaborate. Our users can set up automated reporting and insights to be sent to tools like Slack or Teams. By putting data into a collaborative space through automated reporting, people are able to discuss implications, align and drive action.

Thanks, Britton!

Find and connect with Britton on LinkedIn and Twitter. Check push.ai demo.

Thanks for reading, everyone. Until next Wednesday!

This month paid subscribers learned about:

• Introduction To Event-Based Analytics - How to manage and leverage event-based data for analytics. The best practice of setting up events, properties, and attributes for user activity tracking, and how not to get lost in the event data noise of toggles, hoovers, and scrolls.

• How To Get Average Logins Per User Per Day in SQL - A continuation of the user engagement analysis series. How to get the average number of logins, actions, transactions, or likes per user per day and segment it on paid, in-trial, and free user types.

• Introduction To Problem-Solving And Critical Thinking: nailing problem-solving and informed decision-making. How to develop critical thinking and strengthen your analytical intuition. Common methods and types of analysis.

Subscribe now

This newsletter is a little special ⭐.

Once per year, I reach out to my network of analysts and data experts with one question - what is the best advice you received that changed your career? Today I wanted to share some of the wonderful responses I got. I hope they will inspire all of us to grow in analytics and get empowered with data.

What Is the Best Advice You Have Ever Received?

Kyle Poyar, Operating Partner at OpenView, the author of the Growth Unhinged newsletter: 

Look for ways to replace yourself. Early in your career, you get ahead by being the best analyst -- the person who knows the data better than anyone else. That quickly changes. To truly advance you need to find ways to hand off your responsibilities to someone else. It'll be painful in the short term. You'll feel uncomfortable. You'll need to rethink your work identity. But it's what unlocks your ability to be proactive, rather than reactive, and to point out opportunities to improve that nobody else sees. If you're not uncomfortable, you probably aren't growing.

Timo Dechau, Founder and Chief Data Tool Sommelier at Deepskydata. Author of the How to fix your tracking eBook:  

I would say - that I need to focus. I am too curious, so naturally, I dig into 100 topics at the same time.

What it changed - I picked one topic - tracking setup - and spent my energy on investigating it as far as possible. This brought me to a framework for tracking design and gave me plenty of great feedback when I wrote about it.

I stick to this now, that I try to pick one central topic and write about it. It does not always work out 😁

Chetan Sharma, CEO at Eppo, early data scientist at Airbnb, Webflow. One of the authors of The Eppo Blog - a community for product experimentation: 

Here's a piece of advice that was useful to me, from my good friend Erik Goldman, founding CTO of Vanta.

"When considering opportunities, reduce your decision space to one "got to have it" thing. Decision quality goes down when you simultaneously consider a composite of factors: wealth creation, technical development, adding well-known brands to your resume, people management opportunities. Far better to condense to one goal: "I want to learn about AI" or "I want to start a company after this". For example, before Eppo I knew I wanted to start a company, and thus indexed heavily on proximity to leadership, reporting directly to the CTO."

For the analyst-specific career path, my big recommendations are:

1. Write, both internally and externally. Give lots of presentations. Doing these will grow your brand and credibility, far more than painfully persuading business stakeholders one by one. Writing one external blog post at Airbnb has done more for my career than years of in-house IC work.

2. Think of analyst work as a stepping stone to another functional domain: product, eng, marketing. This isn't to say that you have to leave data work, but the highest impact analysts will be multi-brained with one of those.

Ergest Xheblati - Author of MVSQL Patterns book, author of Data Patterns newsletter:

I remember this one very well. I had joined a tech company and I wasn’t sure what direction to take in my career. I had been working with SQL and doing analytics engineering for a while and I had also worked on quantitative analytics.

I somehow had this notion that I needed to work on quantitative stuff in order to be closer to the business despite the fact that I didn’t enjoy it much. I enjoyed the engineering aspects much more.

I asked a director-level colleague about what he thought I should do and he plainly said: “Do what you enjoy and are really good at. You also happen to be very close to the business through the work that you do so why change careers? The business also happens to value your work as is since they’re already paying you for it.”

That advice was crucial to helping me continue to build my technical skills and eventually led to me writing my SQL patterns book.

Robert Yi - Co-founder and CPO at Hyperquery, ex @Airbnb, @Wayfair. Author of the Win With Data newsletter:  

The best advice I've gotten came from my skip manager at Wayfair, who told me:

"You should always try to work at places that you're really excited to work at." At the time, I remember thinking he was just trying to keep me at Wayfair, but in hindsight, I realized this was fantastic advice. When you enjoy your work, everything compounds. You learn faster, you advance faster, you're more creative. Plus, you're happier, which means more of the rest.”

Chad Sanderson - Head of Data, Data Contracts Advocate. Author of the Data Products newsletter:

Start with the problem, not the solution. Often times data teams get so wrapped up in complexity that they fail to step back and remember the work we are doing is not academic. If you always root work to clearly defining the problem, you will never have to worry about whether your projects deliver value or not.

And reflecting on advice responses I received last year:

Avinash Kaushik, Chief Strategy Officer, Croud, Best-selling Author, author of the Occam's Razor newsletter:

“Every cutting-edge thing you learned in college will be stale in three years. Hence, take jobs where gaining new knowledge is a key part of being successful.”

Looking back, it has been proven to be true again and again and again. I've done my best in every role to stay close to the real work, and that has forced me to keep learning new skills. The alternative fate is I become yet another director/VP, whose primary success is driven by an ability to suck up or play company politics well. 

Benn Stancil, CTO + Founder Mode, benn.substack:

For me, the best advice I ever got was to find real problems that I cared about answering. 

There are lots of tutorials out there that teach technical skills on toy problems and sample datasets. These problems might teach you a few techniques, but they won't make you a better analyst, because learning to be an analyst requires asking questions, seeing a result, being curious about what that result tells you, asking more questions, and continuing to dig until you uncover something truly interesting and useful. With real data, on real problems that you care about you'll do this naturally; your curiosity will draw you further in. On sample problems, you'll often stop when you get to the answer in the back of the book—which, of course, doesn't exist for most questions you'll want to answer.

Chu-Cheng Hsieh, Chief Data Officer at Etsy, Advisory Board Member @Google:

The best advice is to always ask oneself: "Do you want to be effective or do you want to be right?" This applies to both data and leadership.

Let's start with data.

This advice reminds me to take calculated risks because speed matters in business. Often the cost of having comprehensive data to make the right decision requires lots of work. For example, one can set the p-value threshold at 0.01 but it means that you need to run A/B tests for an extended period of time – sometimes you probably never hit 0.01 because when you extend the experiment period, you have to also consider seasonality, model drifting, etc. And that's why most companies set p=0.05. Even if such a threshold naturally introduces false positive treatment, the choice is much more effective.

Let's talk about leadership. This question can be applied in different contexts. I'm providing one example here. Often we are facing multi-choice options. Say, there's almost no right answer in designing an organization. When I have to introduce an org change, I often favor consulting only key people (senior leaders, my manager, HRBP, etc.) While it's absolutely possible to consult every manager in the org to gather more information, keeping the group small comes with benefits like speed and alignment. It would take months if dozens of managers are involved in the process. On the contrary, it would be a blind decision to make an org decision without considering different perspectives. The question reminds me to seek a compromise between gathering inputs, achieving alignment, and making timely decisions.

Adam Kinney, VP of Analytics at Mixpanel, ex @Twitter, @Google:

The best advice I got was to always ask questions about any request for data, metrics, analysis, etc, until I understand the ultimate decision that the requester is trying to make. All requests like these should ultimately be informing some decision, whether that it is a big one-off decision for the company or an ongoing decision, like metrics that inform a manager on whether their team's execution is on track. When you understand the decisions driving requests, you have an opportunity to figure out how best to inform the decision beyond the specific request and how to formulate it into the format that would be most effective for the decision-makers. It also helps with prioritization since the underlying decisions can usually be sorted for importance and urgency better than specific data requests. 

The impact of doing this in my career has been that I moved from a tactical bit player with limited impact to someone who is driving strategy and has huge impact. It has also helped me coach my teams on how to have a bigger impact, which is generally a difficult thing for even pretty experienced data analysts to figure out.

Alex Justman, Vice President, Digital Resiliency at First Republic Bank:

The best lesson I got from possibly one of the worst managers I ever had “Never be victimized by your circumstances”

You have to own your situation. If things turn against you you need to figure out how to assess the position you are in, and identify who you need to work with to provide a solution to your customers!

Naomi Ceder, Chair Of The Board Of Directors for Python Software Foundation, Author of The Quick Python Book:

35 years ago when I was in my first job as a high school language teacher (in Latin), my first mentor urged me to learn more about computers and programming because he thought it would become important and getting in on it in the early days would be both interesting and good for my career. He was more right that I think he could imagine, since that start everything for me. 

The second advice was from my boss was to accept a posting London to help start a new company about 10 years ago. While there were various hassles for someone older to take a position like that, for me it was a great opportunity to expand my network and diversify my skills, which was valuable in later positions.

Matt Brattin, Founder TMB Analytics, ex VP of Analytics @Aplos Software: 

I have two pieces of advice that came to me at different times which I've never forgotten:

1) Before I got started, I asked an alumni of my graduate program something he wished he'd known before getting started and he said to me "there is no such thing as perfect data". It didn't mean a lot to me at the time, but definitely as I got out into the wild it made much more sense in that the best you can do is understand your data and "respect your data", so that you can handle it with care knowing there is always going to be some nuance to it.

2) Immediately after getting promoted to my first manager role I was told by the head of my vertical "people are unpredictable". This was meant in a way to prepare me for people management and knowing that you can't know everything about the way people work all the time, especially when the team gets large. What I took from this, though, is the importance of communication and creating an environment for openness where you can reduce the unpredictability, but you can never eliminate it fully.

Thanks for reading, and thank you to everyone who shared their learnings ⭐.

• Playbook For Launching, Monitoring, and Analyzing A/B Tests - my step-by-step process of analytical support for full A/B tests lifecycle.

• How To Locate The Right Frequency Of Push Notifications - methods and analysis on how to find the right threshold of notification frequency (or email cadence, ads impressions, payment upsells, etc) to make sure it converts into the highest DAU, while at the same time also doesn’t harm user engagement.

• Getting Date Functions in SQL - a SQL guide on date and time formatting, providing pointers on which Date Time function to use for which question or case.

Subscribe now

Today’s newsletter is a special one. I am excited to invite my second guest writer, a blogger, an analyst, an engineer, and a health and wellness enthusiast - Madison Schott 🥰.

I am way overdue with my coverage of analytics engineering. I keep receiving questions on what dbt is, why analytics engineering has become so popular so quickly, and how we can expect it to continue to change the industry. Should analysts re-qualify now and become data engineers to keep up with the industry demands?

There’s a lot to unpack here. There is no one better to introduce analytics engineering than Madison Schott, the author of the newsletter:

I am very grateful Madison found the time to write a guest piece for us before she continues to stay busy with getting married and celebrating life! 

is an analytics engineer at ConvertKit and the technical writer behind the Learn Analytics Engineering newsletter, wherein she writes about transitioning from a non-traditional tech background to a data role, working with dbt and SQL, and best practices. She is also the author of the ebook "The ABCs of Analytics Engineering" that I highly recommend for anyone looking to dive deeper into the field and learn the skills necessary to transition to this role. When she's not writing about data, you can find her cooking up a meal with ingredients from a farmer's market or on a hike in the sunshine.

What is analytics engineering, how it is transforming data analysis and data engineering, and why it is important

Data analysts work to serve marketing, growth, product, and financial teams by providing them with reports and dashboards highlighting KPIs and other various metrics that help drive business decisions. Data engineers work to capture data from different applications and external sources and deliver that to the data analysts to use. But there is a major gap in these two roles that nobody seems to be talking about. 

I’ve experienced this gap firsthand as an analytics engineer. Data engineers capture the data they think should be captured, never consulting with business teams on what data is most important to them. Data analysts then have to piece together the data they do have, producing a report that may not paint the entire picture. Oftentimes, these two roles don’t know how to properly communicate with one another about what they need due to a lack of technical or business knowledge. This is where the analytics engineer steps in.

What is analytics engineering?

Analytics engineers sit in between the data engineer and the data analyst. They have both technical skills and an understanding of how the business functions. Their primary skills include:

• Data modeling 

• SQL 

• Technical documentation 

• Data pipelining 

• Testing 

A large emphasis on what analytics engineers do revolves around producing high-quality data. 

Many analytics engineers use a data transformation tool called dbt. While dbt isn’t a defining characteristic of an analytics engineer, the company that created the tool is actually the one that created this new role. Most analytics engineers use dbt because of the best practices it instills when data modeling. It allows them to write modular, fully tested, fully documented, and advanced data models. 

dbt compiles SQL code so that the same piece of code doesn’t have to be written over and over again and can instead be referenced in multiple data models. It also uses a templating language called Jinja that makes it easy for dbt to connect to your data warehouse, reading from raw data and writing to your development and production databases. Dbt is also powerful because of its macros, which are essentially Jinja functions that can be used throughout your SQL code. 

How analytics engineers are different from data engineers

The difference between analytics engineers and data engineers is still a nuanced topic. Data engineering has become such a broad term nowadays with all of the different languages, tools, and skills grouped under it. It’s becoming harder and harder to narrow down job descriptions and find what you really need as both an employer and a job candidate. 

We have slowly seen data engineering branch into different types- devops or site reliability engineering, machine learning engineering, and now analytics engineering. This is because the responsibilities and skills are different! They deserve to have their own title to make it easier for everyone in the job market. 

Analytics engineers still have a lot of responsibilities similar to that of data engineers. They are in charge of moving data from external sources to a single source of truth, or the data warehouse. They own ingestion, transformation, and orchestration. The big difference between analytics engineers and data engineers is really that transformation component. They aren’t just capturing and moving around data, but understanding it on a deeper level. 

The transformation component done by analytics engineers requires an understanding of business processes and what that data should look like. Data engineers don’t typically need to understand the data itself, more so just the metadata. Analytics engineers can really bridge the gap between the more technical processes and the characteristics of the specific data being collected. 

How analytics engineers help data analysts 

I like to say that the data analyst is the analytics engineer’s stakeholder. Everything the analytics engineer produces is essentially for the data analyst to then use in their reports and dashboards. Data analysts serve the business and analytics engineers serve them.

Data analysts without an analytics engineer on their team may be writing data transformations, all of which live directly in their BI tool. This not only slows dashboards down but also slows down the whole analytics process. Analysts are forced to standardize data every time they want to write something new and repeat code that they’ve already written.

When an analytics engineer joins the team, their job is to transform the raw data into data models that can then be directly used by the data analyst. The standardization of columns, joins, and whatever other transformations needed are done within these data models instead of the BI layer. This ensures key datasets are always available to be used and contain the highest quality data that has already been validated.

Now, data analysts can focus on using these datasets to simplify the BI layer and produce the KPIs needed by the business. Dashboards and reports can be done quicker and trusted for accurate results.  

Why it’s important 

Data quality is a bigger issue now than ever. We have so much data available at our fingertips, but how do we know it's accurate? What’s the point of using data to drive important decisions like where to spend money on advertising or how to increase customer retention if it’s wrong? 

Analytics engineers make producing high-quality data their top priority. By bridging the gap between data engineers and data analysts, discrepancies between how the data is being captured and what is needed can be better understood. Analytics engineers draw attention to the issues they see in how data is being moved, and how it can be better improved. 

By standardizing data close to the source using a tool like dbt, analytics engineers are ensuring everyone within the company is using clean and accurate data. When transformations occur at the source, rather than within the BI tool, mistakes are minimized and quality issues are caught before that data has a chance to make it to the BI tool.

The same goes for the complex data transformations that analytics engineers write directly within dbt. These can be validated while they are being written, then properly tested, before being deemed reliable and ready to use to make business decisions. Then, they are orchestrated to run on a cadence so that these datasets are always highly available when they are needed, speeding up the entire analytics process! 

If you want to become an analytics engineer…

Hopefully, you’ve gained a better understanding of what analytics engineering is, how it relates to the other data roles, and why it is more important now than ever. I’m a firm believer that analytics engineers are here to stay because of the value they provide for data teams adopting a modern data stack. They help to bridge the gap between data analysts and data engineers, focusing on producing high-quality data for business teams. 

If you think analytics engineering might be the right career for you, I’m here to support you in your journey! Polish up on your SQL skills, follow some dbt tutorials, and continue reading newsletters like this one. The first step is to learn as much as possible, the second is to put that knowledge to work. 

Thanks, Madison! 

Find and connect with Madison on LinkedIn and Twitter.

Thanks for reading, everyone. Until next Wednesday!