Subscribe now

A quick note: I have a 20% discount for my readers for the upcoming dbt Summit in Las Vegas on Sep 15–18! If you work with dbt, analytics, or ETL, this is a must-attend event for anyone working with data. Expect breakout sessions, hands-on labs, and time to connect with fellow analytics engineers.

Hope to see you there! Use code Ext-OlgaB at checkout.

In early June, Anthropic published a guide - How Anthropic enables self-service data analytics with Claude, and the analytics world was never the same.

For a few weeks, the post was everywhere - in Slack conversations, internal discussions, and all over my LinkedIn feed.

I know it’s been 2 months since that article was published, and many people have probably forgotten about it and moved on. Well, I haven’t.

This was one of those rare historic moments when an AI lab officially confirmed something data teams have been saying for so long: you cannot simply point a model at a warehouse and expect reliable answers.

It admitted that out of the box Claude never achieved more than 21% accuracy in answering analytical questions, even after verifying Claude had read all the materials and content available. In another test, even after providing Claude with all pre-existing analytical content - all the SQL queries, notebooks, dashboards, etc, it improved accuracy by only 1%.

So what did? Data governance, encoding analytical workflows and business context as skills and human validation. Anthropic keeps human sign-off on anything reaching leadership and tests against golden datasets.

It basically confirmed you need a village of analysts to make self-service work and keep it going.

After yet another piece of LinkedIn cringe that made very little sense, I decided to feature an Anthropic article today to (a) celebrate its unusual transparency and (b) level set expectations for where analytics stands and why so many self-service initiatives will likely fail.

Feel free to forward this to your CEO if someone still thinks your data team is slow and behind.

What Anthropic proved

Anthropic clearly said - connecting Claude directly to a data warehouse through MCP gives it access to the data, but access alone does not make its answers trusted or reliable. Claude still does not know which of several similar fields contains the approved metric, which business rules it should apply, or whether a table was retired last quarter. MCP does not teach Claude how the company defines revenue, customers, activation, or churn.

Anthropic’s data shows that typically models perform only at 21% accuracy when first introduced to a new business domain. It eventually can get improved up to 95%, but only after a senior data team organized the business context, corrected mistakes, and continuously tested the system against a known set of questions. That means, 95% is not the expected accuracy of AI analytics out of the box. It takes a village to get there.

In Anthropic’s setup, the semantic layer provides trusted definitions the AI needs before it queries the data. The problem with that is most companies do not have a semantic layer, and building one can take ages. And if you already have one, supporting, maintaining, testing, and verifying it also takes resources.

Connecting an AI model directly to the warehouse (and this is what most of us are doing now) is not enough. Reliable analytics still requires shared definitions, human review, testing, and ongoing maintenance.

I love this summary from Ayush Gupta, Genloop CEO:

Anthropic’s self-service analytics post is the clearest public case any AI lab has made that you cannot point a model at a warehouse and get reliable answers. Five thousand words in, they say it directly: pointing Claude at a warehouse “can create a false sense of precision.” The rest of the post is the infrastructure required to undo that. It is the most honest public account of what production-grade agentic analytics actually demands. It is also a near-perfect illustration of why most companies will not be able to follow the same path.

The team behind “lean” self-serve analytics

Let me start with this: the company that wrote about how it successfully implemented self-serve analytics had at least 30 full-time data scientists and analysts (plus an unknown number of data and ML vendors). And it is hiring more, clearly aiming to expand its analytics team.

That may not seem like a lot for a large company. At the time, those 30 roles represented approximately 2% of Anthropic’s employees. This puts Anthropic in the same range as Meta, Google, Grammarly, and Amazon, where data roles also represent around 2% of total headcount. All are known as best-in-class, data-driven companies that invest heavily in data and analytics across tools, culture, and decision-making.

In other words, Anthropic may describe itself as lean and efficient, but it has still made a significant bet on data and analytics - at the same relative scale as MAANG companies.

Interestingly, compare that with the share of data analytics roles at fast-growing companies: Lovable at 4%, Duolingo at 5%, HubSpot at 6%, NVIDIA at 7%, and Perplexity at 8%. It almost looks as if rapid company growth might actually be connected to… having more analysts in-house. 😈

And honestly, with 30 data scientists and analysts, you can probably build a rocket, launch it using AA batteries, and produce a dashboard proving that the landing was within an acceptable margin of error.

Jokes aside, what makes data and analytics roles different (and expensive) is that much of its value is reusable. A trusted revenue definition, customer model, or experimentation framework should work for a team of 5 and remain correct for a company of 5,000. More users and more complexity certainly create additional maintenance, governance, and support work, but the requirement does not necessarily grow one-for-one with company headcount.

If you want self-serve to operate with a tolerable level of accuracy and confidence, it does not begin with an AI assistant. It begins with significant investment in data infrastructure and data talent.

Successful self-serve starts with investment in data infrastructure

Garbage in, garbage out - only faster

In analytics and BI, you get what you pay for. Investment in data governance, data quality, metadata, lineage, and business definitions goes a long way when companies try to build self-serve analytics or integrate AI into their systems.

You can’t build real intelligence on top of incomplete 3rd party data, disconnected tools, or knowledge sitting in people’s heads. Sooner or later, it all comes back to the importance of having multiple backend services, in-house user resolution and identification, a strong data management platform - and that still requires a strong team of data experts.

To be fair, garbage in, garbage out has always been a problem. But now, with so much AI-generated content and code, garbage gets created 10x easier and 100x faster.

That is why companies should invest not only in AI tools, but more importantly - in the analytics infrastructure behind them.

Teams need systems that help them validate test setups, trace where events come from, pull a list of impacted users, review user sessions, and connect dots between different data sources to catch problems before they spread.

More on getting the analytics foundation right:

• When To Use Client-Side Or Server-Side Data

• Introduction To Event-Based Analytics

• Do You Over-Report DAU?

• How To Set Up Analytics for Web and Mobile Products

Integrations are where self-serve gets really expensive

An average small company now uses between 10 and 50 data sources. An enterprise may use more than 1,000. I didn’t believe that number either, so had to look it up.

To build reasonably mature analytics or offer reliable self-service, companies must connect these sources, resolve user identities, and map the data to consistent business definitions. This is extremely difficult. No one does it seamlessly, which is why there is an entire generation of tools (CDPs) solving with varying degrees of success.

You already know my opinion on those (How To Make A Sandwich in 587 Steps).

A typical mobile app may use 15-30 different marketing and growth tools, usually connected through SDKs, APIs, or server-side integrations. Each tool can introduce its own user identifiers, metric definitions, attribution rules, and formulas. Teams eventually have to maintain 20 versions of the same KPI, or worse - one massive Excel spreadsheet trying to reconcile all of them.

I see nothing wrong with a large spreadsheet if it serves the team well. Modern spreadsheets can be dynamic, cloud-based, directly queryable, and even provide some version control. And that worked reasonably well in the dashboard era. It is not enough in the agent era.

AI agents need governance that is systematic, executable, and available wherever the work happens. The agent needs governed access to the relevant data, shared definitions, and a reliable way to connect users, transactions, events, and sessions across systems.

That does not necessarily mean you have to copy everything into one warehouse. You can make data and business logic accessible through connectors and map them to the same identity and semantic framework. But the underlying problem remains the same for all of us: can we accurately identify the same customer across 50 different tools and connect that person to the correct transactions, events, and sessions? No AI model solves that.

Reliable and trusted analytics needs expert owners

Let the right people own the right work

I guess I belong to the annoying old-school group that still believes relevant experience matters. Probably because some of the biggest messes I have seen were created when people without the necessary expertise were asked to own and maintain particular stacks or systems.

At an early-stage company, it is normal for a tech lead to handle data + security + infrastructure + reporting. It works to a degree, and at that stage, it is often the only practical option. The problems begin when the company grows and teams continue building on top of that temporary foundation.

There should be a special form of punishment for:

• Data analysts owning production data engineering.

• Software engineers defining data models and business reporting.

• Product managers creating “napkin math” for financial metrics and revenue forecasts.

• Marketing managers interpreting A/B tests.

• Data engineers or DBAs tuning ML or optimizing models.

People can and should work across functions, especially on small teams. But critical systems and projects (building a data warehouse, implementing event tracking, establishing reporting and experimentation, designing data models, and creating forecasting or financial estimates) need clear owners with the right expertise.

Many teams initially optimize for cost and speed by being “lean” and removing short-term bottlenecks. Later, they pay for that through immature analytics, unreliable reporting, and broken trust in their data:

• Contentsquare’s broken analytics

• Lack of trust in reporting at Airbnb

• Inconsistent insights and lack of trust in reporting at Uber

• Expensive and slow data processing at Razorpay

For Airbnb, the semantic reset took almost 3 years. At Uber, adoption of the new metrics system took at least 18 months. A full transformation (migrating data, rebuilding models and BI systems, and achieving company-wide adoption) can take a few years.

Even with the right infrastructure and owners, self-serve runs into one more limitation: the tool can retrieve data, but it can’t decide what the business should be asking.

AI can’t ask the right question for you

I’m convinced that fully self-service is a myth. In my experience, it mostly fails, but not because of data or accuracy- it fails because of (a) questions asked and (b) semantics.

Stakeholders don’t ask “How many MAU we have in France” or “What’s our net new revenue on iOS?”. That would be easy to solve, and most mature companies have these stats easily available.

Instead, they ask:

• “If we increase X by 2x, how much will Y improve?”

• “What can I change in A, B, or C to generate an additional $200K next month?”

• “If we sunset features X and Y, how much of a hit to retention should we expect?”

These questions require more than retrieving a number. They involve identifying relationships, making assumptions, estimating unknowns, and often building a model. They also require someone who knows whether the question can even be answered with the available data.

Unfortunately, Claude will still give you an answer, even when the necessary data does not exist, the assumptions are wrong, or the underlying data is unreliable. That means users must be able to question the result, challenge its assumptions, and validate the output. Not every stakeholder can (or should) be expected to do that.

This is why a large part of an analyst’s job is reframing the question. The conversation often ends with either, “We can’t answer that with the data we have” or “We can’t answer A, but we can provide B instead”.

Now imagine that the let-me-help-you-ask-the-right-question layer is gone. What happens?

I once spent more than 4 months building an internal querying tool. It allowed business teams without SQL knowledge to pull data for marketing initiatives, media profiles, OKR estimates, and whatever. The goal was to free analysts from daily, repetitive ad hoc requests.

The outcome was that teams could retrieve the data themselves, but they needed even more of the analysts’ time. They needed help understanding whether the data actually answered their question, whether the numbers were correct, and whether they were comparing apples to apples or apples to oranges. On top of that, the analytics team became responsible for access management, onboarding, user support, documentation, and ongoing tool maintenance. In short - no, it didn’t free up analysts from data requests.

As Ken Rudin, former head of User Growth and Analytics at Google and former head of Analytics at Facebook, put it more than a decade ago:

“I would rather have great people using poor technology than great systems and a team of people who don’t know how to use data. In the end, big data is only as effective and impactful as the people you have to analyze it.”

+100 to every word - and it’s even more relevant in 2026.

Thanks for reading, everyone!

Related publications:

• The Guardrails AI Needs for SaaS and Subscription Reporting

• What “Normal” Looks Like: A Guide to Baselines

• Claude for Excel: What It Does Well - and Where It Still Falls Short

• Amplitude Agents 101: How They Work + 20 Prompts to Try

• Figuring Out Chaos. Now What?

• Who Owns Growth? (And Why Analytics Pays the Price)

Today, I’m going to try to prove 3 claims:

1. You probably don’t need a semantic layer if you have strong dbt models. Well-designed marts should provide enough business meaning for both analysts and AI agents. Think twice if you really need another abstraction layer on top of it.

2. dbt models can be enough - but only *if* an analyst built them. Or, better - if they were designed by someone who understands the business questions they need to answer, and not only the origin of source data or ways it needs to be transformed (this is where too many analytics engineering projects stop).

3. Which brings me to: your data lead needs BI instincts (even if BI is not in their title). The person leading your data initiatives should have spent enough time in BI to know how people use, combine, and misuse data and potential damage from it (e.g., over-reporting MAU is annoying but not as harmful for the business as under-reporting churn).

Read below on the difference between semantic layer vs data model vs metrics layers, with examples for each, and how to set up dbt to serve both analysts and AI agents well.

Why we can’t agree on what semantic layer is

Semantic layer is the most misunderstood concept in data. Many people think of it as an aggregated layer of predefined metrics and formulas built for BI tools. But it is (or at least can be) much more than that.

One reason we can’t agree on its definition is that the term means different things depending on the data stack being used. For example, Power BI or Looker has a built-in semantic layer that provides a layer of security, defines metrics, and implements hierarchies. In this case, the semantic layer and the metrics layer are combined. But other data stacks separate the transformation layer, semantic layer, and metrics layer, and then there is a new generation of tools designed to sit at different stages of data processing.

For example, a few weeks ago, wrote a wonderful piece How AI revived semantic layers and the quest for self-serve analytics. It is one of those must-read takes on the modern data stack, but once again, the history of semantic layers is presented as a Looker supplement, when in fact it’s so much more.

I think of a semantic layer as a standalone, warehouse-centered product that can serve multiple domains. It may be driven by BI needs, but it can also support ML, security, integrations, APIs, and operational workflows.

Even more, reducing the semantic layer to a BI-only (which is a common setup) is dangerous. Many BI tools don’t have version control or a formal deployment process, there is lack of visibility into definitions. Which means, you likely won’t be able to reproduce the same metric in another tool. This makes it harder to scale, creates multiple sources of truth, and leaves different systems reporting different answers to the same question.

Semantic layer vs data model

Read more

A few months ago, I wrote about how to approach product analytics when your users become AI agents or when you offer AI products: When AI Agents Become Users: Rethinking Analytics Tracking.

My main message was that traditional product analytics still works, but it no longer tells the full story. We can track whether a user completed onboarding, activated, or converted. But between onboarding and conversion, an AI product may make many decisions based on the user’s behavior - how to personalize the experience, which recommendation to provide, or how to respond.

These decisions aren’t reflected in a typical product analytics report: which model was used, which prompt shaped the response, what the agent remembered, and why the system selected that particular configuration.

So I proposed a new tracking framework to make this layer visible with a set of configuration, exposure, and outcome events.

That may sound simple and straightforward, but once you implement it, it opens Pandora’s box:

• What should count as an exposure if the user never experiences the selected AI model?

• What should we measure when the intended setup fails and the system switches to a different model or configuration?

• Should we evaluate the experiment by user, session, task, or request? What if the AI setup can change throughout the experience? (This is my biggest challenge now to figure out.)

• Did personalization improve the results, or did it just reach users who were already more likely to succeed?

• Is a cheaper model still cheaper if users need more attempts (or more time) to complete the same task?

So, this is what Part 2 is about. It is relevant to product analysts and data scientists responsible for setting up analytics, event tracking, measuring AI features and quantifying their impact on business outcomes.

When setting up tracking, my rule has always been that less is more: there is no need to overwhelm the analytics system with noisy events that provide little or no value.

However, with AI, we are still figuring out what works, what doesn’t, and, more importantly - what success looks like. To be safe, I am leaning toward capturing more rather than less.

For my analysis, I need visibility into the model setup, the input provided, the configuration selected, and what was executed. I need request-level granularity so I can measure and report on what the system intended to do, what it actually delivered, and whether it moved any meaningful product or business metric.

The decision layer is becoming part of the product

I call it the “decision layer”, but I’m also seeing people call it router or orchestration layer. A router selects a path or configuration. An orchestration layer manages the workflow. I use the “decision layer” as a broader term for the logic that decides which model, prompt, tools, context, and validation a request should receive.

Basically, it is the layer that sits between the user experience, the AI configuration, and the analytics system. It reads live signals about the user and the request, then decides which model, prompt, tools, and context the agent should receive.

Why we need it: Most AI products do not process every request using the same fixed configuration. Depending on the question, the decision layer may:

• Use a fast, lower-cost model to retrieve a governed metric from the semantic layer.

• Use a more sophisticated model with access to SQL, experiment assignments, tracking releases, and data-quality tests.

• Return a cached answer when the underlying data has not changed.

• Query live warehouse data when the answer needs to be current.

• Use a specialized tool or model for forecasting, anomaly detection, or customer segmentation.

• Run additional validation.

• Reconcile revenue against an internal billing system before returning the result.

• Switch to another model or tool if the original configuration fails, times out, or produces a low-confidence answer.

To the user, these may all look like the same AI assistant or product. However, underneath, the product may follow a very different path for each request. Those decisions affect the answer’s accuracy, cost, latency, and ultimately the user’s behavior. So your team needs visibility into those decisions and the ability to quickly read and iterate them.

BTW, this is exactly the reason why feeding AI more context isn’t always a good thing. You need to be intentional and effective with it.

Setting up AI performance tracking

Read more

If you missed the July posts, here’s the roundup:

• How Much Is an A/B Test Worth? - A new framework for deciding when to test, estimating how much each experiment is worth, and optimizing A/B tests for business value rather than sample size and statistics.

• Should You Discount to Save the Customer? - Not always. Napkin math for estimating whether to offer a renewal discount and how low to go.

• The Guardrails AI Needs for SaaS and Subscription Reporting - How to build a subscription waterfall that gives AI the guardrails it needs to keep churn, revenue, and subscriber metrics aligned and accurate.

Subscribe now

I’m so excited to feature Tristan Handy today, founder of dbt Labs and now President of the combined Fivetran + dbt Labs, which serves more than 100,000 data teams worldwide.

We met at the Databricks Data + AI Summit in San Francisco, where I (finally!) had the chance to interview him. We talked about how AI is transforming analytics engineering, what data teams need to do to adapt, how Fusion and the agentic development loop fit into dbt’s vision for the future, and more.

I assume everyone reading my newsletter knows dbt well, but if you don’t - dbt is a data transformation platform that helps data teams turn raw warehouse data into reliable, documented, and reusable datasets using SQL and software engineering practices (version control, testing, and automated deployment).

For analytics, dbt was (and still is) transformational. It helped establish an entirely new role - analytics engineering (read more - Introduction To Analytics Engineering)

Read below Tristan’s take on why analytics engineering may matter even more in the AI era, and why the future of the role depends less on writing SQL and more on building the context, structure, and trust that AI needs to work reliably.

For years, analytics engineers created value by writing SQL, building reliable data models, and bringing software engineering practices into analytics. But now that AI can write SQL, does that make the discipline less relevant?

Tristan’s answer is counterintuitive: analytics engineering is becoming both less relevant and more relevant at the same time.

The code itself is becoming less visible. An analyst may ask an agent to build a model without ever touching the SQL underneath it. But the principles behind analytics engineering, like defining business logic, testing it, documenting it, and keeping it under version control, are becoming more important because AI agents need those structures too.

AI may reduce the value of writing complex SQL by hand, but it does not reduce the value of understanding how a company’s data fits together, and that understanding defines the next era of analytics engineering.

dbt was a workflow breakthrough

I have followed dbt since its early days. What it eventually became is difficult to overstate. I don’t know of a data team today that does not use it. Everyone does. It is now so embedded in analytics work that I truly have no idea how we did it all before.

To me, analytics engineering exists as a distinct discipline largely because of dbt. However, when Tristan and his co-founders started building dbt, they were not trying to invent a new profession.

They put into practice the idea that analytics should be treated like software, where SQL models were modular, changes were reviewed in git, logic was tested and documented, and analysts could work with data using practices already common in software engineering.

The approach made their consulting team more effective, but it took 3 or 4 years before the rest of the ecosystem began to pay attention. According to Tristan, the shift happened around 2019 and 2020, when dbt went from “a couple of people in a little Slack community” to becoming “part of the canon.”

For me, the reason why dbt received such massive adoption was by turning a good way of working into a system that people could adopt. As Tristan put it: “dbt is a product that encodes a set of best practices. It is not a technology breakthrough. It is a workflow and best-practice breakthrough.”

Making analyst-written code safe

I was initially skeptical of encouraging analysts to write production code. Analysts come from very different backgrounds: product, finance, marketing, operations, economics, and many others. Giving everyone unrestricted access to write transformations without shared standards can create a spectacular mess.

However, dbt made that work safer and more structured. It gave analysts conventions, testing, documentation, lineage, a framework for collaboration, and most importantly - the operating system around the SQL.

AI changes who writes the code, but not why it matters

This brings us back to AI.

Tristan agreed that one part of the original analytics engineering role is losing value: the ability to write complicated SQL manually.

“The analytics engineering skill of understanding data systemically is still important, but specifically being able to write complex SQL is less valuable than it used to be.”

dbt itself initially required users to know SQL, Git, the command line, data warehouses, and enough query optimization to understand why a model was slow. But those barriers are getting lower now. An agent can write a model, explain an error, generate a test, and refactor downstream references. The person directing it may never need to write the final SQL.

But the code still has to exist.

An agent can work with a dbt project because the project’s logic is represented in files it can read and modify. Definitions, tests, dependencies, and documentation are available as structured context. Business logic trapped in a spreadsheet, a sequence of clicks, or someone’s memory is much harder for an agent to use reliably.

“Agentic development has made it even more obvious that everything needs to be defined in code.”

In a way, this is the paradox - AI makes code less important as a user interface, while making code-based systems more important underneath.

The analyst may stop typing SQL, but the organization cannot stop defining what its data means.

The next data problem is context

Every major data and AI company is now talking about context. This is partly because models have improved faster than the information companies can provide to them.

The model can generate a valid query, but it still does not know:

• which revenue field finance trusts

• whether a cancelled annual subscription remains active until the end of its term

• which test accounts must be excluded

• why 2 dashboards use different definitions of an active customer

• whether a table is current, deprecated, or safe for executive reporting.

These are organizational knowledge problems, not SQL-related.

Tristan believes the next challenge is to create a shared context layer for structured data and make it available to everyone asking business questions through tools such as ChatGPT or Claude.

The goal sounds straightforward:

A person asks a question about a business metric, the request passes through a governed context layer, a query runs against the company’s data platform, and the person receives a trustworthy answer.

Technically, most of the pieces already exist, but in practice, very few companies have assembled them well.

Version control, testing, and modular SQL all existed before dbt. dbt’s contribution was turning those pieces into a workflow that data teams could actually use. Tristan sees the context layer at a similar stage: the components are available, but the operating model is not yet settled.

The Fivetran–dbt Labs merger makes this direction clear: the companies completed their merger in June 2026, bringing data movement and transformation closer together. They also announced Agents Schema, an open-source proposal for storing metric definitions, semantic models, lineage, and business documentation in SQL tables that agents can query.

Instead of being a feature hidden inside one BI or AI tool, this is a bet that the semantic and context layer will become infrastructure.

Context is also a cost problem

Giving an agent access to more information does not automatically make the system better. In fact, it can make the system slow, expensive, and noisy.

Tristan used customer-call data as an example. A company could connect an agent directly to a system such as Gong and ask for feedback about a product. The agent may pull many full transcripts into its context window, consuming thousands of tokens for each call. Or the company could load the call data into its warehouse, use dbt to clean and summarize it, and give the agent a smaller, structured dataset.

Tristan estimated that this kind of preprocessing could significantly reduce token use for the query. The exact savings would depend on the transcripts, models, prompts, and summary design, but the underlying point is that raw access is not the same as efficient context.

We already learned this lesson with analytics. Giving everyone access to raw production tables - nothing good came from it and it did not create self-service analytics. Teams needed cleaned models, shared definitions, and governed access.

AI is repeating the same cycle at a much higher speed and cost.

Agents should not search every system of record from scratch for every question. Some context needs to be ingested, modeled, summarized, tested, and reused. Companies will also need to decide which tasks justify expensive reasoning models and which can run on smaller, cheaper ones.

So the context layer is doing 2 jobs:

1. Improving accuracy by giving the agent trusted definitions and data

2. Controlling cost by reducing how much irrelevant raw information the agent processes.

The goal is not to give agents as much context as possible. It is to give them the smallest amount of trusted context needed to answer the question reliably.

Core, Fusion, and the agentic development loop

dbt’s recent product changes show how the company is adapting its original idea for this new workflow.

dbt Core v2 is the 2nd major version of Core and is currently in alpha. It replaces the Python foundation with a much faster Rust-based runtime and remains open source under the Apache 2.0 license. Core and the Fusion engine now share that foundation.

Fusion adds richer SQL comprehension and development features on top. Instead of treating a SQL file largely as text, it can understand columns, types, and downstream dependencies. That allows the development environment to catch syntax and type errors earlier, support more precise column-level lineage, and help refactor references when a field changes.

These features improve the human developer experience, but they may matter even more for agents, because an agent is more useful when it can see the impact of a change, validate its work, and receive a precise error before it sends an expensive or incorrect query to the warehouse.

The current rollout still has caveats. Support varies by adapter and environment, and dbt’s documentation still lists some adapters and capabilities as preview or beta. Teams should evaluate actual compatibility rather than assuming every Core project can migrate without testing.

Will chat replace notebooks and BI tools?

When I asked Tristan about analytical notebooks such as Hex and Deepnote, he gave a more provocative answer.

“I think that notebooks are great, but I think that entire experience may just get sucked up into ChatGPT and Claude.”

His argument is that a chat transcript already resembles a notebook. In a traditional notebook, the analyst writes code and the kernel returns an output. In a chat interface, the analyst writes in English and the model produces the code, analysis, or chart.

I personally disagree - there is still a lot that needs to happen before we get there.

AI can create a chart, but in 90% of cases it creates one that I would not show to an executive - labels are poorly placed, categories are sorted incorrectly, colors do not communicate meaning, axes are confusing, and so on. The chart may be technically valid and still fail as communication. It takes me too many iterations with AI to plot something usable before I give up and do it myself.

Tristan acknowledged that BI is “incredibly hard and competitive and thankless.” A polished dashboard requires control over every small detail (which may help explain why dbt historically stayed away from competing directly with Tableau or Power BI).

Interestingly, during our conversation, he mentioned an experimental dbt Charts prototype: dashboards defined in YAML, rendered as HTML, and authored with the help of an agent. It’s not another drag-and-drop BI tool, but rather storing dashboards beside the dbt project so that the definitions, models, and presentation layer can be changed through the same development workflow.

Good visualization remains hard. With AI, the interface may move from SQL editors and notebook cells into chat, but the durable assets underneath - models, definitions, tests, and even dashboard specifications, still need structure.

What becomes more valuable for analysts

If AI can write SQL, analysts should be moving one level up:

1. Understanding how the business operates

2. Translating ambiguous concepts into precise definitions

3. Designing data models that can answer more than one question

4. Identifying which source should be trusted and where it breaks

5. Building tests and reconciliation rules around important metrics

6. Evaluating whether an agent’s answer is plausible

7. Communicating the result in a form someone can use to make a decision.

SQL still matters. You need enough fluency to inspect generated logic, recognize a bad join, notice double counting, and understand why a query produces the wrong grain. But writing every line manually is no longer the scarce part.

The scarce part is knowing what the query should mean.

This also changes how I think about the future of analytics engineering. The role expands from organizing transformation code to organizing business context for both humans and agents.

Analytics engineers helped companies move logic out of scattered dashboards and into tested, reusable data models. Their next job may be to move organizational knowledge out of scattered documents, meetings, SaaS tools, and people’s heads into a governed context layer. That is much harder than generating SQL, but also much more valuable.

AI agents need the guardrails dbt was built to provide

The original dbt idea that analytics should be modular, tested, documented, and version-controlled, becomes even more important with AI.

When humans wrote every transformation, weak documentation and inconsistent conventions produced slow, fragile analytics. When agents generate transformations at machine speed, those same weaknesses can produce bad logic faster and at a much larger scale.

Agents need constraints, definitions, lineage, and tests. They need to understand what a change will break downstream. In other words, they need the principles analytics engineering introduced all along.

AI may write more of the code, but someone still has to define what the data means, decide which logic to trust, and build the system that keeps it reliable. That is why analytics engineering is expanding beyond code and into the context that both people and AI need to make trustworthy decisions.

Thank you, Tristan!

You can find Tristan at:

• Linkedin

• X

• The Analytics Engineering Roundup (Substack)

• The Analytics Engineering Podcast

You can learn more here:

• Learn more about dbt and its products

• The dbt Developer Hub

• The dbt Fusion engine: ship dbt at the speed of AI development

• dbt certifications

I’ll explain how recurring reporting should be set up for SaaS vs B2C subscriptions and share the framework I use to keep churn, ARPU, revenue, sales, and active subscriber metrics accurate and aligned.

I’ll also show you how to task AI with monitoring these metrics, identifying inconsistencies, and maintaining that alignment over time.

AI cannot fix a reporting framework built on the wrong assumptions.

SaaS and B2C subscriptions may share the same basic lifecycle, but their billing systems, customer volumes, plan structures, and definitions of churn and active subscribers are very different. Before giving reporting to AI, you need to account for those differences - or AI will simply automate the same inconsistencies faster and with more confidence.

SaaS reporting frameworks don’t work for B2C subscriptions. We use them anyway.

For reference, here is how I classify the companies discussed in this publication:

1. SaaS: Asana, Slack, QuickBooks, Figma, Zoom, and Atlassian.

2. B2C Subscriptions: Calm, Duolingo, Strava, Netflix, Spotify, Disney.

Do not confuse them with:

• B2C Transactional (one-off payments): eBay, Amazon, Temu, Uber, Airbnb.

• B2C Social (Ads): Twitter, TikTok, Facebook, Instagram, YouTube, Messenger, WhatsApp.

• Enterprise SaaS: Salesforce, Workday, IBM, Oracle, Adobe, and Microsoft

• Large enterprises: Wells Fargo, Kaiser Permanente, Mercedes-Benz, and government agencies such as the FBI

This publication focuses only on the first 2 categories: SaaS and B2C subscriptions.

Here is a shocking reveal: B2C is very different from B2B.

At first glance, if you think about a subscription lifecycle (New Subscription → Renewal → Churn → Winback), it looks quite similar to SaaS. So, it seems logical that typical SaaS reporting (Start Period customers → New customers → Churn → Net New Customers, all tied into ARR) should work.

And it mostly does - with some adjustments. But things start to go wrong once you introduce more payment providers, more plans, or you need to report on expansion and contraction.

I run into this problem a lot: fast-growing B2C companies often bring in interim CFOs and COOs with traditional enterprise SaaS backgrounds. CFOs expect subscription reporting to follow the same rules they already know, and they make my life miserable when it does not. And I struggle to explain that YouTube or Netflix reporting should be different from analytics at IBM or Mercedes.

I spend a lot of time explaining that the concepts are similar, but the mechanics are not. Churn works differently in B2C, ARR is often the wrong anchor, and even something as basic as an “active paid customer” may require a completely different definition. If those differences are not addressed, every downstream metric, from ARPU and revenue to retention and subscriber counts, can quietly drift out of alignment.

If you need help explaining to your CFO why a 2% reporting accuracy threshold won’t work for your app (either mobile app or web B2C subscriptions product), send them this:

Why SaaS reporting breaks in B2C subscriptions

1. Quicker user lifecycle:

The average subscription retention for mobile apps is 3-4 months. In B2C and especially in mobile apps, users cancel and switch between plans far more frequently than in SaaS, where annual contracts or longer billing periods are the norm.

Some billing systems record plan changes by ending one product ID and creating another. In 90% cases, without additional logic that you apply with SQL, upgrades and downgrades are misclassified as churn followed by a new subscription, inflating both metrics.

2. Too many SKUs or product IDs:

In SaaS, you typically work with 2-4, maybe 6 plan types:

It’s simple to hardcode these plan movements in your data layers. In mobile apps, however, you could easily be dealing with 80 product IDs or 40 SKUs. In one extreme case, I had to deal with 248 unique product IDs and report on user movements (upgrades and downgrades) between them. Don’t do it. Nothing good comes from it.

3. Self-serve vs. guided onboarding:

In SaaS and B2B, customer onboarding is often assisted by account representatives who guide customers through each lifecycle stage, including renewals and upgrades. As a result, reporting is often handled manually. There may be no concept of a “signup,” and reporting is typically built at 2 levels: the organization level and the individual customer level.

Your most important metric is beginning customers, which typically serves as the denominator for most other metrics.

In B2C apps, tens of thousands of users move through standardized onboarding flows and paywalls. Your analytics must account for skewed distributions and large numbers of outliers, and every stage of reporting needs to be automated.

Your most important starting metric is signups or installs, which serves as the primary denominator for most other metrics.

The expectation for reporting accuracy is also different.

In SaaS, you are expected to keep discrepancies below 2%. And that is often easy:

You may have fewer than 10K paying customers, with a small number of “whales” on expensive annual contracts. You can practically count them on your fingers and validate every customer ID across the 2 or 3 systems - Salesforce, Stripe, and monday.com. Everything is reconciled in one large spreadsheet everyone knows about, and financial forecasts can be precise and clean.

In B2C subscriptions, a 20% discrepancy can feel like a good outcome:

You may have more than 500K paying customers spread across weekly, monthly, 3-month, 6-month, annual, and lifetime plans. Some come with a 1-week trial, others with 1-month. Some have no free trial. Payments come through Apple’s App Store, Google Play, Stripe or even PayPal, each operating as its own processing system and database.

There is no single, consistently reconciled source of truth. RevenueCat, Chargebee, Baremetrics, Recurly, and Adapty all show different numbers. Calculating the total number of active paid subscribers in your database requires a long SQL query, and the result still differs from the number your CFO has.

Even after months of cleaning and reconciling the data, you still don’t know whether your “active subscriber” count includes free trials. In theory, it should not. But then why is it so high?

If you’ve been there - bookmark this publication. The framework below will keep your sanity and metrics aligned.

One framework that keeps subscription metrics aligned

I have trust issues. I don’t trust the reporting in RevenueCat, ChartMogul, Adapty, or Baremetrics. I have worked with subscription data for 10 years, and I have yet to see churn rate or LTV reported accurately by a 3rd-party tool.

So, when I start working with a new app that offers recurring products, the first thing I do is build a subscription waterfall. I try to build one separately for iOS, Android, and Stripe, and then map all payment providers together.

This gives me a baseline for where the metrics should be, compared with what 3rd-party tools report. It also helps me establish the range of natural variation in each metric. Later, I can give those thresholds to an AI agent and have it flag unexpected deviations.

The concept of waterfall (or your 5-min MBA crash course)

Read more

What often happens next is that the team targets the customers identified as “about to churn” and offers them a discount to encourage renewal.

The next challenge is determining the right discount threshold: how low can we go to retain the customer with the least possible revenue loss?

In this publication, I’ll walk through the framework I use to estimate that threshold, along with how to approach the underlying modeling and reporting.

Before we start, let’s align on metrics and definitions

It all comes down to NRR

Net Revenue Retention measures what happens to revenue from the same group of existing customers over time.

It reflects:

1. Upgrades and expansion

2. Downgrades and contraction

3. Renewal discounts

4. Cancellations.

Important: it excludes revenue from newly acquired customers.

How to read it:

• Above 100%: existing customers generate more revenue than before.

• At 100%: expansion offsets all losses.

• Below 100%: the existing customer base is shrinking.

Why a discount is important: A customer moving from $100K to $80K creates $20K of contraction. That is better than losing the entire $100K, but it is still a revenue loss.

The problem with renewal discounts

Teams tend to overvalue saved renewals. They see it mostly as logo (customer) retention: “We discounted, therefore we saved the account.” But it’s very difficult to model what would have happened without the offer. The customer might have:

1. Renewed at full price

2. Accepted a smaller discount

3. Reduced scope but stayed

4. Delayed the decision

5. Churned regardless of the discount.

First of all, no churn prediction model is perfect, and any model can be wrong for an individual customer. This framework relies on predicted probabilities, so probability calibration is especially important, and it requires ongoing maintenance. You should also monitor precision, recall, lift by risk band, and model stability over time.

Second, the problem with discounted renewals for the “about to churn” cohort is that even if a customer accepts it, they’re still more likely to churn during the next renewal cycle than customers not in this cohort. Even after renewing, these customers may remain at higher churn risk during the next renewal cycle, especially when the discount does not resolve the underlying product, budget, or value problem. Their future expansion potential may also be lower than healthier accounts.

Finally, teams often over-estimate renewal math: the customer was worth $100K. We renewed them for $80K. Therefore, we saved $80K!

This calculation assumes that without the discount, churn probability was 100%. But that is almost never known:

• If the customer had a 70% chance of renewing at full price, then the expected retained revenue without the discount: 70% × $100K = $70K

• If the discount raises renewal probability to 80%: 80% × $80K = $64K

So the company saves more customers but generates less expected retained revenue.

This is why this is difficult, and needs a village of data scientists to model it right. At the end of the day, it all comes down to retained revenue.

My framework for modeling discount threshold

For every proposed renewal discount, I model 2 things:

Read more

They also know (and I take some pride in that) that A/B tests must be conducted properly and follow clear statistical protocols. I have spent years writing about the importance of statistical significance, statistical power, and p values. I have also warned teams that even when they do everything right, still, 70%-92% of A/B tests will be inconclusive or misleading - showing an apparent lift when the true effect is neutral or even negative.

This is why many scientists and analysts remain skeptical of A/B testing:

I used to be in that group too.

Many teams do not invest in proper experimentation, and they test everything. Inaccurately. Then, they become convinced that shorter onboarding flows perform better than longer ones, hard paywalls increase conversions, or dark mode improves engagement, all based on that one test they ran 2 years ago. They keep sending me a slide / case study with 2 numbers for Control and Variant, showing some % lift. I’m not able to replicate it, validate it, pull those users to check distribution, or do anything with it, and yet I’m supposed to treat it as a proven causal learning? (Onboarding tests are particularly good examples - they’re the hardest experiments to conduct, and if you’re measuring them using typical events setup via Amplitude or Mixpanel, I’m confident your test read is wrong.)

I am less skeptical of A/B testing now because I have seen what experimentation looks like when it is done at scale. Best-in-class companies often run the same test many times across different traffic levels, seasons, audiences, budgets, and product conditions. When teams can separate the effect of the change itself from the effects of seasonality, audience mix, spend, or other conditions, they can be much more confident that the result is real and repeatable.

This is one reason companies are now doubling down on accelerated experimentation, scaling from 10 tests per month to 100 or more:

If most tests don’t give a trusted answer, teams need an environment where they can quickly identify the few tests that produce clear results and act on them.

But scaling experimentation creates another problem - cost.

Once teams begin running experiments at scale, things become chaotic. They often cannot tell whether they are running too many tests, too few tests, or tests that are much larger and more complicated than necessary. They simply plug in common statistical rules like 80% power and p < 0.05, without connecting those decisions to revenue, cost, or business risk.

What it means is that some tests can run for months. They take traffic away from more urgent or valuable experiments and slow down the entire testing program. A team may increase from 10 tests per month to 100, yet complete fewer than 2.

Methodologically, the team may be doing everything right:

• You cannot stop a test before it reaches the required sample size or statistical threshold.

• At the same time, you cannot realistically adjust the confidence level for every experiment - accepting 90% confidence for one test, 85% for another, and 95% for another, when hundreds of tests are running.

So what should teams do?

Data scientists and researchers from Amazon, MIT, Stanford, and Columbia collaborated to solve this exact problem. A few months ago, they published a paper asking: when companies scale experimentation, how should they decide how much each experiment is actually worth? In other words, how do you optimize A/B tests for business value rather than sample size and statistical conventions?

Below, I will walk through their research and explain how their framework helps teams make experimentation decisions based on economics rather than statistical rules alone.

Read more

If you missed the June posts, here’s the roundup:

• Omni: The Better Looker, or Just Another Expensive BI Tool? - A deep dive into Omni’s semantic layer, BI-as-code workflow, customer feedback, AI features, and where the tool still falls short.

• 30 Must-Read Books to Become a Better Analyst or Data Scientist - My collection of books that I believe are essential for becoming a stronger analyst or data scientist, develop analytical intuition and critical thinking.

• The Rise of the AI Product Analyst - What AI Product Analysts do, what skills they need, which tools are must-know, and how analysts can prepare for a role that sits between product analytics, data science, analytics engineering, and AI product development.

Subscribe now

Last month was… interesting.

First, there was Snowflake Summit, with over 190 booths and more-or-less the same ETL tools everywhere. They even started to sound the same: Datavault, Datalab, Daasity, Daemon, Daman, DataArmy, DataClymer, DataIQ, and 100 more.

Then came Databricks Summit, which was probably the best event I’ve ever been to - both in scale and content. More than 31K attendees! That’s bigger than my hometown.

First of all, I had a chance to meet and interview Ali Ghodsi, CEO & Co-Founder at Databricks. I also joined the launch of a new book, Databricks Data Intelligence Platform by Jason Yip and Marcin Woztyczka. Very good timing, because I’m wrapping up the AI Agent Databricks certification, so it was helpful to have a step-by-step breakdown of the Databricks ecosystem. The book is easy to follow, and I recommend it to anyone working with Databricks products. It’s still free to download, but hurry, it probably won’t stay free for long.

I also interviewed Tristan Handy, CEO & Founder at dbt Labs (my interview is coming soon, and it’s sooo good!). I also spoke with the Omni team (yes, they saw my Omni review. Yes, it was a little awkward), and I spent time with the Hex team.

This newsletter reflects my recent conversations with leaders from Databricks, Informatica, and Salesforce about AI hype vs AI’s real impact on analytics and BI.

Below, I want to demystify a few AI myths affecting data and analytics. And this time, the discussion is finally backed by early AI benchmarks that show the gap between leadership ambition - becoming “AI-native”, pushing everyone toward “AI engineering”, and demanding AI-driven work, and the actual state of things inside companies.

Before I continue, I want to acknowledge how fortunate I am to speak with and learn from the people contributing to this publication - Ali Ghodsi - CEO of Databricks, Manouj Tahiliani - SVP of Salesforce, Gaurav Pathak - SVP Product Management AI and Metadata at Salesforce, and many others. Years ago, I would not have imagined that a personal analytics newsletter could become a bridge to conversations with some of the strongest practitioners and builders in data and AI. I’m grateful for the access, the learning, and for all of you who continue to read this somewhat private journal on where analytics is going. Thank you for being here ♥️

Data platforms are shaping analytics

I liked Databricks Summit this year, and I also enjoyed Informatica World a few weeks earlier, because both conferences opened with transparency and a reality check:

1. Garbage-in is garbage-out: your data quality is the foundation for your AI.

2. AI maturity and AI success directly depends on your data management: your ability to maintain metadata and context. And context comes with a cost - often a tremendous cost, and there are no shortcuts.

That is exactly what I have been arguing for a long time. In analytics and BI (especially in analytics and BI!), you get what you pay for. Investment in data governance, data quality, metadata, lineage, and business definitions goes a long way when companies try to build self-serve analytics or integrate AI into their systems. You cannot build real intelligence on top of incomplete 3rd data, disconnected tools, or knowledge sitting in people’s heads. Sooner or later, it all comes back to the importance of having a strong data management platform - and that still requires a strong team of data experts.

Myth 1: Everyone is using AI

No, they are not. Everyone is talking about AI: many companies are buying AI tools, launching pilots, adding copilots, and telling employees to become “AI-driven”. But that is not the same as AI being embedded into how work gets done.

The Databricks benchmark report shows a perception gap:

CTOs are much more optimistic than the people closer to execution. 89% say AI rollout is ahead of schedule, but only 77% of VPs agree.

And even when companies say they are adopting AI, the underlying work often has not been redesigned enough to support it:

Only 37% of firms say they have fully redesigned processes across the organization. Another 38% say multiple functions have redesigned processes. That still leaves a lot of companies layering AI on top of old workflows, old incentives, old data problems, and old governance gaps.

Myth 2: Better model, better answers

There are so many models now:

One of the common myths is that analytics, BI, and self-serve data get better as models get smarter. It sounds logical - if the model can reason better, write SQL better, and summarize better, then the answers should improve, right?

Wrong. It is often the opposite. Smarter models expose and break over the company’s missing context, messy data, weak governance, while skyrocketing the cost:

A model can reason only over the context it receives. If the company’s data is fragmented, definitions conflict with each other, no metadata, and source-of-truth logic lives nowhere, the model is failing. The more advanced model, the more it relies on the context.

This is why the semantic layer conversation is so back everywhere now. AI makes missing semantics very painful. If someone asks for “customer data,” a public model does not know whether “customer” means a company, an individual, a B2B account, or a B2C buyer or particular user_id. That meaning lives in the catalog, the glossary, the MDM layer, and the semantic layer - not inside the model by default.

So AI is likely to fail not because the model is not smart enough, but because companies are not organized enough for powerful, sophisticated, and expensive models to understand them.

Myth 3: Ten agents are cheaper than one human

The idea that companies can replace repetitive human work with agents and automatically save money is dangerously incomplete.

Agents can reduce some human work, but they also create new machine work. They read more data, call more APIs, query more systems, create more logs, generate more code, trigger more workflows, and require more monitoring. The cost problem is massive data usage. And agents are designed to increase that usage.

The benchmark report confirms that for AI agents, cost and resource constraints are one of the biggest barriers to scaling. Every autonomous action creates data that may need to be logged, stored, monitored, and sometimes audited. Agents also require more computation than traditional generative AI, which adds more cost.

Agents need data access, permissions, context, identity management, monitoring, observability, lineage, evaluation, cost controls, and governance. Many companies underestimate how much they consume infrastructure:

This also changes how we should think about AI code generation. If AI writes much more code over the next year, that code still needs to be stored, reviewed, secured, versioned, governed, maintained, and eventually deleted. Otherwise companies will create a new layer of technical debt faster than they can understand it.

Agents put real pressure on infrastructure. If usage triples, the cost goes up, even if the users are agents and not humans. Companies need to think about this now. They need to invest in context management, model routing, filtering, and systems that can separate useful output from noise.

Cost is already going through the roof for many teams. To control that cost while still investing in AI means knowing which models to use for which tasks: simple models for simple work, heavier models for important or high-risk work. Agents need that context too.

Myth 4: Accuracy is static

If you ask a model a reasoning question and it performs well, it is natural to trust it the next time. That is one of the biggest AI dangers: one accurate answer does not guarantee the next answer will be also accurate.

AI accuracy is an dynamic condition, but many think of it as a fixed product feature.

The quality of AI output has to be maintained over time to address changes in data, business rules, definitions, permissions, models, user behavior or systems:

“About three in five firms review AI systems during development and before deployment. Fewer than two in five continue that oversight after a system goes live—the stage where AI models drift, data shifts and edge cases multiply. Worse, one in eight firms reviews governance only when something goes wrong.”

Anthropic says agentic analytics accuracy drifts 95% → 65% in a month without maintenance. But maintaining model quality output is not something teams scope time for, but this one is critically important: a model that summarizes a meeting incorrectly is annoying. A model that changes a customer segment incorrectly is expensive. But an agent that takes action based on outdated, incomplete, or unauthorized data is a governance incident.

This is why AI quality needs to be maintained like a production system. Teams need recurring evaluation, post-deployment monitoring, data freshness checks, drift detection, incident review, escalation paths, and analysts who are trained to question the system.

Gaurav Pathak from Informatica made a good point about governance: “Many companies start by focusing on building the best agent, but governance has to be considered at the beginning of the cycle. Otherwise agents may take private customer information or brand-sensitive information and make it available to people who should not see it.”

This is where many companies are still stuck in an old governance mindset ,where governance is treated like a document, a checklist, an Excel sheet of approved models, or a wiki page of certified or golden datasets. That worked in the dashboard era, but it is not enough in the agent era. Agents need governance that is executable, systematic, embedded, and available where the work happens.

Myth 5: More agents means more impact

More AI code or agents does not automatically mean more AI impact, like more pilots do not mean more value.

Based on benchmark report, high AI activity can hide very thin returns:

“In 2026, high levels of AI deployment mask thin returns. Our survey finds that more than four in five executives say their AI programmes are beating expectations. Yet only about two in five firms formally require teams to track business impact, from cost savings to revenue and efficiency”

That means many companies are busy with AI, but far fewer can prove that AI is changing the business.

You can count AI-generated code, prompts, pilots, tools, dashboards, copilots, and agent workflows. But none of that proves business value. It should go down to what changed:

• Did costs go down or did cycle time improve?

• Did revenue go up? Or did customers get a better experience?

• Did employees actually save time, or the company just created another tool they have to manage?

From my interview with Gaurav Pathak, professionals still spend around 80% of their time getting data into the right shape - cleaning it, preparing it, governing it, and making sure it is fit for purpose.

From the benchmark report, a few examples how teams quantify AI impact:

Stellantis cut its AI portfolio to 20 programs, each required to show measurable value within 12 to 18 months. Suncorp started with 120 AI ideas, narrowed them to 20, and killed several that could not justify their cost. One of Suncorp’s selection rules was reuse: build once, deploy many times.

Myth 6: Tools or tech determine whether AI scales

No. Culture does.

The hardest part of making AI work is rewiring the organization around them: redesigning tasks, training people, changing workflows, creating the right incentives, and making it safe (and encouraging) for employees to question AI output.

The benchmark report points to a major mismatch:

“Half of firms cite human review as a top ongoing cost, yet only 4% point to employee upskilling. Firms are wrong to think that they can keep AI running without investing in the people who must work alongside it”

Changing work is harder than changing job descriptions.

This is why I keep coming back to data platforms and data teams.

AI makes weak data work more dangerous. In the dashboard era, bad data created bad reports. In the agent era, bad data can create bad decisions, bad customer actions, data leaks, compliance failures, and uncontrolled cost.

The future of analytics depends on whether companies can build and maintain trusted context around those models, which means better governance, matured semantic layers, cost controls and monitoring, and yes - better data teams.

Thanks for reading, everyone! Until next Wednesday!

Related publications:

• The Rise of the AI Product Analyst

• Amplitude Agents 101: How They Work + 20 Prompts to Try

• What AI Terms Mean for Analytics

• Before You Build an Analytics Agent

• When AI Agents Become Users: Rethinking Analytics Tracking

• Anticipating 2026: Top Trends in Analytics

• Figuring Out Chaos. Now What?

• Claude for Excel: What It Does Well - and Where It Still Falls Short

• The Biggest Data, Analytics, and AI Conferences of 2026

But even with more demand, finding a strong product analyst is still hard because the role itself is still evolving.

Expectations for product analysts have also changed a lot over the last few years. It is no longer enough to produce insights, support experimentation, or set up tracking. The role is now starting to blend with analytics engineering and ML. Historically, product analysts often had support from data scientists and data engineers for data modeling, testing environments, and statistical analysis. Today, that support layer is very thin. More often, analysts are expected to own much more of the product and data lifecycle.

Meanwhile, AI is changing what products look like and how they need to be measured. Product teams are now building conversational interfaces, AI agents, recommendation systems, enrichment workflows, and generative features that do not behave like traditional static product flows. At the same time, analysts are being asked to use AI tools to diagnose metric changes, automate recurring analysis, and explain user behavior faster.

That means the Product Analyst role is emerging in 2 directions:

1. The first is building analytics for AI-driven products: measuring whether AI features are useful, trustworthy, accurate, and valuable.

2. The second is using AI for product analytics: applying AI-native tools and workflows to analyze traditional products faster and more effectively.

In this publication, I’ll break down both sides of the role: what AI Product Analysts do, what skills they need, which tools are must-know, and how analysts can prepare for a role that sits between product analytics, data science, analytics engineering, and AI product development.

Why AI makes product analytics more important

I’m sure you saw this chart too:

As I predicted a year ago, AI acceleration is bringing harsher competition for downloads, high mobile store rankings, market share, and, most importantly - customers. Success ultimately depends on understanding customer behavior and product usage. Only one discipline truly owns that knowledge and context - analytics. Demand for analysts will remain high, especially for product analysts.

Product analytics now mostly falls into 2 distinct categories:

1. Building analytics for AI products, which includes evaluating AI models and user experiences

2. Using AI for product analytics, which means leveraging AI to analyze traditional products faster and more effectively.

The 2 sides of the AI Product Analyst role

These two sides are connected, but they require different skills, responsibilities, and ways of thinking.

Role 1: Building Analytics for AI-Driven Products

For the past decade, the product analytics playbook has been built around clicks, funnels, and conversion flows:

Before a product launch, teams would design tracking systems to capture key user actions, which later would become metrics (signup started, trial started, checkout completed, feature used, subscription canceled, etc) to follow a simple framework: instrument the user action tracking → translate it into a metric → measure whether the product change moved that metric.

But AI products are different.

Read more

We’re not as lucky as some other fields. There are many books on data engineering and data management. There is even more literature on marketing, finance, and product development. But when it comes to analytics, the list is surprisingly small.

A large part of analytics books falls into academia and heavy statistics. That is still important, of course, but it is often not quite relevant to the work analysts actually do every day. There are very few books that sit at the intersection of data, statistics, and business or product.

Below is my collection of books that I believe are essential for becoming a stronger analyst or data scientist. I’ve read each of them, some more than 3–4 times, and I hope this list helps you find a few useful reads too.

Product and marketing analytics

1. Product Analytics for Data-Driven Decisions: Derive Insights from Web Analytics Data by Joanne Rodrigues.

2. Freemium Economics: Leveraging Analytics and User Segmentation to Drive Revenue by Eric Benjamin Seufert. An old but priceless book that helped set my own journey into product analytics. Still one of the most practical books on segmentation, monetization, and freemium business models.

3. Actionable Gamification: Beyond Points, Badges, and Leaderboards by Yu-kai Chou

   

   A great collection of frameworks for streaks, points, progress loops, rewards, and all the “gaming” mechanics apps use today.

4. Hooked: How to Build Habit-Forming Products by Nir Eyal. A good read for understanding user behavior, habit loops, engagement, and how successful companies build products people keep coming back to.

5. The Cold Start Problem by Andrew Chen. A great book on how network effects start and scale. This is not a typical analytics book, but it is very relevant for analysts who support growth at marketplace or network-based products.

Read more

Last year, I published a guide comparing Tableau vs. Power BI. As I said then, I find both tools overly complex and very expensive - neither would be my choice. That post sparked discussions, with people asking: if not Tableau or Power BI, then what?

Well, there are now more than 200 active BI tools on the market now. More than half emerged after the AI boom, and most of them look like this:

I tested about 8 of these tools (the ones that offer free trial) using my medium-sophisticated dataset and a few questions beyond the usual “How many transactions did we receive in Canada this year?” None of them met the bar. Not even half of it. The visualization quality is a horror story for another time. Most of these tools are simply LLM-to-SQL wrappers, not LLM-to-semantics. They become useless once you move beyond trivial “how much” or “where” questions.

I do not consider these post-AI analytics tools to be BI. So my “best BI” list still focuses on more foundational drag-and-drop, dashboard-building tools like Looker, Domo, Tableau, Power BI, Sigma and others. That is the category I use to evaluate Omni. Not the new wave of ask-anything-and-get-a-pile-of-stats-in-seconds.

Omni has been on my review list for a long time. One of my clients is an Omni customer, so I use it myself. But I am not a typical user. I am comfortable with most BI tools, and my learning curve is usually short (unless it’s figuring out aliases in Tableau. It’s been 12 years... please send help). That is why I also interviewed 11 active Omni customers and spoke with 14 Omni users who either build or read reports. I researched setup and maintenance costs, compared notes across teams, and built a good view of how Omni stacks up against other BI tools in 2026.

Below is everything you need to know about one of (still) new, fast-growing, and most talked-about BI tools this year: a deep dive into Omni’s features, where it performs best, where it falls short, and when it makes sense to use it to justify the cost. And, most importantly, whether Omni lives up to its promise of becoming the best BI tool.

How it all started

I think the way a product enters the market matters. And personally, I did not find the “We built Looker, so we can do it again” story very appealing. But that was enough to raise a $9M+ seed round.

My first introduction to Omni was in late 2023, when Jamie Davidson reached out and positioned Omni as an upgraded version of Looker. So I knew a new BI tool was coming. But the positioning did not appeal to me at first. There was no clear “we do this one thing better than anyone else.” The message was mostly “we are better than Looker.” So what? Lots of tools are.

Shortly after that, I was part of a large migration from Looker to Omni. The data stores were a mess, the semantics were fairly complex, and the migration took about 6 months. That was my first time working with Omni - and it was surprisingly smooth. Their team was involved every step of the way. Their approach to data layering, semantics, and scale was very solid. I could see back then that this team was thinking big.

Long story short, and one acquisition later, Omni now has more than 200 customers and recently raised a massive $120 million Series C at a $1.5 billion valuation!

If that sounds high, compare it with Sigma’s $3B and Tableau’s $15B valuation. That is high. Especially Sigma. Why would someone pay $125K a year for what is basically a spreadsheet?

But back to Omni. With this level of funding, Omni is clearly moving toward the enterprise market and is going after Looker, Tableau, Sigma customers to set itself as the best and most matured BI tool out there.

But is it though? Or is it just another very well-funded BI company with a better origin story?

What Omni gets right - and where it falls short

Read more

If you missed the May posts, here’s the roundup:

• How Asana Found $40M in Recoverable Revenue - How product analytics at Asana turned false churn from user removals into a $40M product decision. Guest post by Kuber Jain.

• When AI Agents Become Users: Rethinking Analytics Tracking - How to set up analytics for AI agents: tracking conversations, testing prompts, and optimizing products in real time.

• 10 Data Concepts Analysts Use Every Day - Part 2 - another batch of 10 concepts every analyst should know. The first 10 are here.

Subscribe now

Hello from Snowflake Summit. If you’re here, I’d be happy to meet. If not, I hope to see some of you next week at Databricks.

May was a busy month for data and analytics. Between conferences, product launches, announcements, and industry events, there was a lot to follow. In May alone, I had 21 events on my calendar, so this recap is a bit longer and more opinionated than usual.

I’ll cover highlights from Mixpanel MXP 2026, Mobile Apps Unlocked (known as MAU), and the Informatica Summit. I’ll also share the key updates from last month across data science and analytics, including new case studies, product announcements, and tools shaping the field, so you can stay reasonably informed without pretending you read every launch blog.

🔊 Advocating for analytics: The new normal and how to figure it out

Mixpanel is ready for the next chapter

I wanted to share a few thoughts on Mixpanel’s “new life.” I call it that because Mixpanel has a new CEO, recent acquisitions, a new strategic direction, and a product that has been changing quickly. I have been using Mixpanel for more than 14 years now! Even though I probably mention Amplitude more often (mostly because Amplitude has been more vocal), I am a more confident Mixpanel user. And with its recent AI direction, Mixpanel currently feels ahead in agent-native analytics.

Overall, MXP 2026 felt like a success. I’ll cover some of the sessions in more depth soon, but here are my main takeaways:

• Companies have shifted from the mobile era into the AI era, and analytics has to adapt. It starts with how we set up taxonomies, catalogs, and events - all need to become native to agents and code, not just to analysts and product managers as before.

• Today, prototyping, development, testing, and design are all moving faster than ever. But speed amplifies bad decisions. Existing product models are collapsing, and we need new frameworks that no one has fully figured out yet. But what we do know is that deep product intelligence is becoming non-negotiable for growth and scale, and that is what teams should invest in. (I really liked how Mixpanel summarized my 318 publications into one sentence).

• Mixpanel is positioning itself as a data layer that teams can build on. It helps separate signal from noise, create a clearer understanding, and compound organizational knowledge. In other words: it wants to become your personal product analyst. Godspeed.

Overall, I agree with Mixpanel’s stance and vision, even if the ambition is high. I also really like Jen Taylor. Her energy is impressive, and given the recent announcements and acquisitions, I think Mixpanel is making the right moves.

It was also nice running into Daniel Schmidt, co-founder of DoubleLoop, the metric trees product acquired by Mixpanel. We may disagree on how practical metric trees are, but I still think they are a unique and smart addition to Mixpanel’s product.

What happened to Mobile Apps Summit?

Mobile Apps Unlocked (MAU) used to be one of the most important events for mobile app growth, AdTech, and MarTech. This year, though, it didn’t feel as strong. I was already disappointed by the event last year, and this year felt even weaker.

This is supposed to be the largest mobile app AdTech and MarTech conference, with more than 2,000 apps attending, at least as claimed. But the quality of the content (and speakers) felt very low.

The exhibition hall was full of marketing and advertising vendors, but the event lacked real-life examples of how these tools and strategies actually work: implementation costs, pitfalls, success stories, and measurable outcomes. Where are the success stories?

For example, I was hoping to learn how web-to-app performs for apps, but why is no one sharing any data on this? I know FunnelFox says it works, but why can’t anyone prove it? I don’t think the math for web-to-app works. I suspect the platform friction plays a bigger role, and I don’t believe the ROI that they claim. I would love for someone to prove me wrong and lay out the math on how investment in web-to-app makes sense. I don’t think it does.

This year, as with last year, there were no strong case studies from app founders or practitioners. There were no real discoveries or lessons backed by data points or benchmarks. I don’t trust broad Phil’s claims like “we’re seeing this everywhere” or Vahe’s “I helped apps generate $50M in incremental revenue through paywall testing” without seeing the data behind them. Almost every speaker seemed to be either from an agency or representing a tool.

To be clear, this is not the case with every event. I always enjoy RevenueCat’s events. For example, Ladder shared its entire product and marketing strategy on stage last year, or how Yu-kai Chou presented a practical framework for gamification (which we actually implemented at MyFitnessPal, and it worked), or how Flo shared their retention framework at the recent Adjust event. I wish MAU were that kind of place where we could upskill, compare data, and share learnings, instead of being an overwhelming platform for promotion. I suspect that after this publication, I may be banned from MAU next year, haha.

Context is a new currency

I didn’t have high expectations going into an enterprise data and AI conference in Vegas. To my surprise, Informatica World 2026 was more practical and on point than any other data event I’ve attended this year (including Snowflake Summit, where I am now).

This was my first Salesforce event (Salesforce acquired Informatica last year), and I was impressed by the scale, organization, and content.

It almost felt like the speakers had read my Before You Build an Analytics Agent publication and decided to dedicate the entire conference to it, with a focus on the importance of context, semantics, data quality, and governance.

I liked how transparent conversations felt - as AI agents become more common, data without context is not enough. Speakers framed context as the new currency of data. It’s the combination of metadata, governance, lineage, quality, and business meaning that allows AI systems to understand and act on data. Informatica positioned itself and its products as the foundation for trusted data, automated governance, and data engineering.

The team announced Headless Data Management, where metadata, quality, governance, MDM, lineage, and policy enforcement are treated as services that AI agents can call directly through MCP and their CLAIRE Agent. They launched Agentic Multidomain MDM - a way to apply agents to the manual work of data cleaning and enriching.

I learned that PepsiCo already has more than 1,500 agents supporting internal systems, Hearst emphasized the growing risks around data leaks and poorly grounded outputs, and Google Cloud highlighted the move from reporting to autonomous agents that can reason and act.

The next phase of data management is the context layer that makes data trusted, usable, and safe for AI. This is where all our attention and investment should be, and I can’t agree more.

🔥 May highlights

Statsig joined OpenAI. No, wait - Statsig joined Amplitude!

There has been some confusion around what happened to Statsig. Initially, OpenAI acquired Statsig in a $1.1B all-stock deal. But now, Amplitude announced a “strategic partnership with Statsig”, under which the Statsig brand, platform, and customers are going to Amplitude. That makes the deal feel like a split between talent and platform. OpenAI wanted the people and experimentation expertise. Amplitude needed the experimentation platform.

My favorite analyst is celebrating 20 years of blogging!

Avinash Kaushik’s Occam’s Razor just turned 20. If you’re a marketing analyst (or really any kind of analyst), reading Occam’s Razor is a must. Avinash started the blog back in 2006, and it has been one of the most practical and influential analytics blogs ever since. I’ve been a subscriber for more than 10 years now, and it shaped the kind of analyst I became. It is still one of the best examples of how to write about analytics in a way that is useful, clear, and grounded in real work. Highly recommend.

In other news:

• Omni makes news again with acquiring Fabi.ai to make a push into headless BI, CLI-driven analytics, MCP, and AI-agent workflows.

• dbt launched Core v2, which now serves as the open-source foundation of the Fusion. That does not fully end the Core vs. Fusion debate, but it means that dbt is keeping an Apache 2.0 open-source Core while Fusion will be the main product direction.

• Amplitude launched llmtrack.org - a public AI-search visibility tracker. It measures how brands show up across GPT, Claude, Gemini, and Google AI Overviews “using more than 44,000 prompts across 30+ categories and 270+ brands”.

• For time-series analysis, check PyTrendy - a new package for identifying and analyzing trends in time series:

📈 New industry reports and benchmarks

1. The State of AI in the Enterprise from Deloitte

2. The 2026 agentic AI readiness index from Fivertran

3. Crosscheck: Benchmarking AI Models in the Real World by LinkedIn Labs

🤓 Analysis and case studies

Powered ≠ Trustworthy by Johan Rydberg (Spotify/Confidence): Spotify wins 12% of experiments but learns from 64%. It’s not all about counting winners.

Causal Inference Is Different in Business by Alejandro Alvarez Perez: Not every decision deserves the same rigour. Match your causal inference to the gravity of the decision: constructive decisions need speed, final decisions need precision.

Experiment Velocity by Jakub Linowski: Jakub created this interactive simulator showing how test volume compounds into cumulative lift. More experiments mean more winners get a chance to ship.

🎓 New tutorials

• Spotify’s Experimentation Bootcamp is now free

• Teaching <Fundamentals of Machine Learning>

• Data 100: Principles and Techniques of Data Science from UC Berkeley, Spring 2026

❤️ Favorite publications last month

Bookmarked to re-read favorite takes

1. Is logistic regression regression?

2. What Every Experimenter Must Know About Randomization

3. Requirements analysis: catching requirement bugs before they become code

4. Notes From the Field: AI, Energy Shocks & the End of the Old Playbook (Spring 2026 Edition) from

5. What data science is actually about in the age of AI from Eric Ma

6. How The Heck Does Shazam Work? from Shri Khalpada

✈️ Data + Analytics events in June

• May 31-June 5, Bengaluru, India: SIGMOD/PODS Conference

• June 1-4, San Francisco, CA: Snowflake Summit 2026

• June 9, New York: Chief AI Officer NY

• June 10-12, Bangkok, TH: Mixpanel MXP

• June 12, London, GB: Research and Applied AI Summit

• June 15-18, San Francisco: Databricks Data and AI Summit 2025

• June 16-17, Sydney, Australia: Gartner Data & Analytics Summit 2026

• June 16-17, London, GB: Data Science & AI Summit

• 23 Jun, online: Causal Summit

Also, check this calendar for the full list of data and analytics events this year: The Biggest Data, Analytics, and AI Conferences of 2026.

Thanks for reading, everyone!

This is Part 2.

This time, I’m focusing on another set of concepts that always show up in analytics work. Most of these come from statistics, ML, economics, or decision science. If you touch analytics, my expectation is that you already know these concepts. If you don’t, keep this list handy.

Most wrong analyses do not happen because someone forgot how to write SQL. Analysis fails because the analyst looked at the wrong population, trusted the wrong metric, overreacted to noise, ignored uncertainty, or found a pattern that was not really there.

So here are 10 more concepts every analyst should know.

1. Stein’s Paradox

Stein’s Paradox is one of the reasons why I’m obsessed and fascinated with statistics. It’s the strangest idea that makes impossible possible. It is also hard to explain.

Basically, it says that when you are estimating several things at the same time, you can often make better estimates by pulling each individual estimate slightly toward the overall average.

This feels wrong at first, but here is an example to explain it better:

If you want to estimate the conversion rate for each country, it seems natural to calculate each country’s conversion rate separately. If you want to estimate retention for each acquisition channel, it seems right to calculate each channel separately. If you want to estimate revenue per customer segment, it seems natural to calculate each segment separately and so on.

Stein’s Paradox says that this separate-estimate approach is often not the best option when you have many noisy groups.

Why? Because small groups are noisy.

A country with 40 users and a 20% conversion rate may not truly be a high-converting country. It may just have a small sample. A segment with 25 users and high retention may not be a great segment. It may just be random luck.

Instead of trusting each small estimate as fully independent, Stein’s Paradox points toward shrinkage: move noisy group-level estimates closer to the overall average. The smaller or noisier the group, the more it should be pulled toward the average. The larger and more stable the group, the more it can stand on its own.

A common mistake is to rank groups by raw averages. This often pushes tiny groups to the top and bottom because small samples are more volatile. A country with 10 users and 4 conversions will show a 40% conversion rate, but that does not mean it is better than a country with 10,000 users and a 12% conversion rate.

Stein’s Paradox explains that the most extreme numbers are often extreme because they are noisy.

2. Monte Carlo Simulation

This is a way to model uncertainty using random sampling. Monte Carlo simulation creates many possible outcomes based on uncertain inputs. By simulating thousands or millions of scenarios, analysts can estimate a range of possible futures.

This is common for cases when direct calculation is hard or when the inputs are uncertain or dynamic. For example, imagine you want to forecast revenue for a subscription product. The future depends on many uncertain inputs: new user growth, trial start rate, trial-to-paid conversion, monthly churn, annual renewal rate, plan mix, expansion revenue, refunds, and more. A simple forecast will use one number for each assumption and produce one revenue estimate. But that estimate can feel more certain than it really is.

A Monte Carlo simulation lets you define a range for each assumption and simulate many possible outcomes. Instead of saying, “Revenue will be $1.2M,” you can say, “Based on these XYZ assumptions, revenue is likely to fall between $1.0M and $1.4M, with a 20% chance of missing target.”

Monte Carlo simulation is used in revenue forecasting, LTV modeling, experiment planning, risk analysis, pricing scenarios, growth planning, and financial modeling.

3. Pareto Principle

Read more

Before a product launch, teams would design tracking systems to capture key user actions. Those actions would later become metrics: signup started, trial started, checkout completed, feature used, subscription canceled.

Experimentation used the same logic. Product analysts had to decide how to instrument each test: where to place tracking, to make analytics cost-efficient and effective, how to measure the lift between control and variant, and whether to send a separate experiment_started event or simply attach the experiment group as a property on an event like trial_started.

This is the analytics world we built: instrument the user action, translate it into a metric, and measure whether the product change moved that metric.

But what happens when users become agents? Even more, what happens when users become conversational, generative, or self-evolving agents that are dynamic and learn from every interaction?

That is where traditional product analytics starts to break.

This publication is about how to navigate and adapt your product analytics setup as we transition from tracking user actions to examining agent interactions. It is about moving beyond basic event logs (which are no longer enough) and treating AI model selection as a core product experiment. That means A/B testing LLMs, prompts, voices, and agent behaviors against key metrics like retention, conversion, and LTV. Also, we’ll look at what analytics looks like for self-evolving products: systems that use real-time data to learn what works, adjust their own settings, and turn insights into action faster.

Product analytics still works. It just does not see the AI layer yet.

Product analytics is still life-essential for ensuring you aren’t getting “garbage in”.

The traffic still comes in. The events still fire. The funnel still exists. Mixpanel, Amplitude, SDK integrations, and all of that tracking are still must-haves.

But the funnel no longer explains the full outcome.

In a traditional product, the page is the product surface. A user sees a button, a form, a paywall, a checkout page, or an onboarding screen. If conversion changes, analysts can usually inspect the interface and look for a visible explanation: maybe the CTA changed, the page layout shifted, the copy was updated, the pricing page was redesigned, or the onboarding flow introduced new friction.

In an AI product, the visible interface may not change at all.

The same user may open the same screen, start the same session, and complete the same flow. But behind the scenes, the experience may be completely different.

The model may be different. The prompt may be different. The voice prompting the prompt may be different. The memory window may be different. The tools available to the agent may be different. The routing logic may be different.

From the user’s perspective, it may feel like the same product. But from the system’s perspective, it is not the same product at all - it could be 42 variations of different setups. The agent itself becomes part of the product surface.

That means agent behavior needs to be instrumented as carefully as user behavior.

The old analytics layer is not enough

In UI analytics, sessions or clicks become events.

We track button clicks, page views, signup starts, card toggles and banner hoovers, trial starts, purchase completions, and cancellations. We use those events to build funnels, cohorts, retention curves, and conversion rates.

That still matters, but in agent-based products, turns or switches become events.

Every conversation turn, model response, tool call, routing decision, voice selection, prompt variant, and memory decision can affect the user experience. This requires a 2nd layer of instrumentation to address agent analytics.

Read more

I asked my good friend Kuber Jain to share a behind-the-scenes look at analytics in SaaS, including how his team at Asana used product analytics to drive a feature worth $40M.

Kuber is now a Senior Data Scientist at Headspace. Previously, he worked on analytics at Asana, DocuSign, and Optimizely, where he helped product and engineering teams use data to improve user experience, engagement, experimentation, and business outcomes.

Below, Kuber walks us through a project from his time at Asana, where his team was trying to understand and reduce customer churn. They segmented users, identified which customers were most likely to churn, studied when they came back, mapped the friction points, and quantified the business impact. Let’s break it down.

Asana is one of the leading work management platforms, helping teams plan, organize, and execute their work. The company grew to more than 170,000 paying customers around the world.

The case study Kuber shares below is about Asana’s Pause Member feature.

I wanted to share it today because the lesson applies far beyond Asana. In SaaS, users often leave for operational reasons: a team changes, a project ends, a budget gets cut, or an admin removes inactive seats. That does not always mean the user no longer needs the product.

Asana’s Pause Member feature is a good reminder that not every lost seat means lost product value. Sometimes, what looks like churn is actually a billing or seat-management problem. In this case, more than 60% of removals were tied to budget pressure rather than true employee turnover, turning what could have been a temporary cost-cutting decision into a permanent product exit.

I’ll hand it over to Kuber to break down what the team found, how they quantified the opportunity, and how those insights shaped the feature.

Why Asana built a “pause member” feature - and why it was worth millions

Below, I’ll walk through the analysis that led Asana’s team to the solution: how they segmented user removals and churn, measured when removed users came back, identified which users and organizations were most affected, mapped the friction in the return path, and quantified the business impact of fixing it.

Before getting into the analysis, it’s worth grounding ourselves in what churn means for a SaaS business, and why not all churn is the same problem.

What is churn?

Churn is when a customer or user stops using a product.

In SaaS, this usually means a company stops paying, which is revenue churn, or a user stops using the product, which is user churn.

Revenue churn hits revenue directly. User churn is often an early warning sign. But in Asana’s case, the team was looking at something more specific than general churn: users being removed from a workspace.

What is the user removal rate?

User removal rate is the percentage of active users removed from a workspace over a given period, usually by an admin.

This metric matters because removal does not always mean disengagement. A removed user may still need the product, but the current workflow gives them no easy way to pause, return, or keep their history. Why does it matter so much for SaaS?

SaaS businesses depend on retention. Even a small improvement can have a big long-term impact. A commonly cited benchmark: reducing churn by just 1pp can increase a company’s valuation by 12% or more over five years, because of how retained revenue compounds.

Types of churn

The Asana case is mostly about false churn. That difference matters because it changes what you should fix.

You do not fix false churn with better onboarding or more support. You fix it by changing how the product handles temporary exits, so users are not treated as permanently lost when they still have a reason to come back.

Problem:

Users weren’t leaving because the product failed them. They were being removed by administrators making short-term budget decisions, and the product was treating each one as a permanent exit. The analytics work was about proving that, quantifying it, and justifying the fix.

The question every product analyst should ask before writing off churn: how many of these users actually wanted to leave? The answer is rarely 100%.

The analysis: 5 Steps that led to Pause Member

The Pause Member feature didn’t come from a product instinct or a sales request. It came from a structured analytical process. Here’s how it unfolded - step by step.

Step 1: Segment the removals to find the recoverable pool

A high removal rate doesn’t tell you much on its own. You need to break it down.

Read more

If you missed the April posts, here’s the roundup:

• Amplitude Agents 101: How They Work + 20 Prompts to Try - An introduction to using AI in Amplitude, including a guide to Amplitude MCP, Global Agent, specialized agents, and the prompts that make them more useful.

• Claude for Excel: What It Does Well - and Where It Still Falls Short - How to get started with AI in Excel, what Claude for Excel does well, where it still fails, and how to use it safely for analytics and finance.

• How to Choose the Right Proxy Metric for an A/B Test - There is no single best proxy metric that works forever. The right proxy depends on the size of your experiment, and the best way to choose it is to learn from your past tests instead of reusing the same metric again and again.

Subscribe now

May is already shaping up to be a very busy month for data scientists.

This week starts with the Data Governance Summit and Tableau Annual Conference in San Diego. Next week, I’ll be in San Francisco for Data Council, Mixpanel, and SaaStr - all happening on the same days, (purposely?), I assume to test our scheduling models. I’ll clone myself.

After that, I’m heading to Las Vegas for MDM + Informatica and MAU, which, of course, also happen at the same time. Hope to see some of you there. Somewhere.

Today, I have the key April updates for you across data science and analytics, including new case studies, analyses, and developments shaping the field - so you can stay informed, relevant, and maybe explain the news to your coworkers before they forward it to you.

🔊 Advocating for analytics: The ROI case for investing in existing users

As someone who was “raised” by high-retention products and apps that build long-standing, frequent relationships with users (think Strava, Calm, MyFitnessPal, Glow, Coinbase, and others), I really struggle to understand the newer batch of apps (AI or not) that mostly monetize through high-volume user acquisition without any real intention of building stickiness or retention.

I don’t want to recycle the same old point made by literally every analyst: durable, sustainable growth comes from retention, not acquisition. But what do you do when many apps don’t expect, and don’t even aim for, long-term usage?

Some apps seem to exist almost entirely to monetize the Day 0 user interaction. They invest heavily in optimizing the first 24 hours of the user experience, and then barely care what happens after that.

My problem with that is that many of these apps are sitting on a massive pool of lapsed users. These are users who signed up, finished onboarding, but failed to activate and never converted. Most never returned after Day 1. Some uninstalled the app, but many others did not. They are simply asleep. They sit in your database, quietly adding storage and processing costs, while the team does nothing to wake them up.

Instead, all the budget and attention go to the next batch of new installs - only a tiny share of whom, maybe 0.24% if you’re lucky, will contribute to whatever monetization model the app has.

For one app I know, the pool of sleeping users is over 350 million!

Think about this: a simple push notification CTR benchmark is around 4%. Even if you get only half of that, you are still reaching a meaningful share of users who already know your product, already installed your app, and already exist in your system. This is not ROI yet, though, a click is not a purchase (but what if you’re running ads?).

Once users come back from a push notification, then you can use in-app messaging, paywall tests, lifecycle offers, and product nudges to move them toward conversion. So add to this the 12% in-app message CTR benchmark to estimate the resurrected MAU opportunity.

I used to operate with a 5% in-app message-to-purchase rate, which is high, as these are woken-up, engaged users. But every app is different - let’s assume a modest-to-conservative funnel:

• 3% push CTR

• 10% in-app message CTR

• 1% purchase conversion after the message click

Such a sleeping-user pool could still generate around 10.5K purchases with zero additional spend. Imagine all that free money just sitting there.

Well, depending on how much Braze costs today… Maybe not exactly free money. But still, way cheaper than buying the same user twice. You already paid to acquire these users. You already store their data. You already have the audience.

It doesn’t matter whether you expect users to leave after signup, or this is a utility, wallet, e-commerce, traveling, or the app designed for occasional, seasonal, or rare usage - the math still works. And yet, instead of trying to reactivate even a small fraction of them, many teams keep pouring money into acquiring more users, 80% at least of which will likely follow the same Day 0 → Day 1 → gone path.

I’m being Captain Obvious here, and most teams understand the math well before I come in to break it down for them with an attitude. The main reason things stay this way is usually not the math. It is (1) a lack of ownership and (2) a lack of developed processes and workflow for re-engagement, which mainly means… a lack of ownership.

Who owns re-engagement? Who owns paid retargeting? Is it monetization, product, or lifecycle? Who is responsible for waking these users up?

Which brings me back to my earlier question: Who Owns Growth? (And Why Analytics Pays the Price). In this case, though, it is not only analytics paying the price. It is the entire company.

Back to new user vs existing user value, if you still need convincing, here are recent predictions from AppsFlyer:

“2026 outlook: As the mobile market matures and UA costs rise…, 2026 is expected to accelerate the shift toward a “Retention First” economy. With Gartner (2025) reporting that 80% of future revenue for mobile businesses will come from just 20% of existing customers, the strategic imperative will lead to greater focus on maximizing LTV of existing users.

As a result, remarketing investment is also expected to increase in 2026. In a stabilizing economic environment where “growth at all costs” is replaced by profitability, reactivating a dormant user will be viewed as more capital-efficient than battling for a new one in a crowded space.”

🔥 April highlights

Google Cloud Next 2026 - what we need to know

Google is trying to turn BI + data platforms into an “agentic analytics” stack, with BigQuery, Looker, Data Studio, and Knowledge Catalog all moving toward governed AI agents:

1. Google released a new “context layer” for AI agents called the knowledge catalog. The goal is to give AI agents trusted metadata, business definitions, semantic context, verified queries, and governed search. It aggregates metadata across BigQuery, Firestore, Looker, Atlan, DataHub, and enterprise systems (e.g., Salesforce, SAP, ServiceNow, and Workday).

2. BigQuery got Graph + AI features: semantic logic moves into the warehouse, new anomaly detection features, and more ways of working with unstructured data.

3. Looker becomes “agentic” with BI agents, dashboard agents, and assistants. They also re-branded Looker Studio back into Data Studio with a new mission: “BigQuery conversational agents and Colab-built data apps”.

Omni has raised a $120M Series C, bringing the company to a $1.5B valuation.

People claiming that AI killed BI are clearly wrong. Omni is showing the opposite, with reported 4x year-over-year revenue growth.

I’ve been working on a separate deep dive on Omni, similar to the ones I did on Superset, Power BI, and other BI. I’m looking to speak with analysts who are currently using Omni and are open to sharing their experience. Please reach out if you’re open to sharing your feedback.

Hex continues to invest in AI, and this time, in visualizations.

As a Hex power user, one thing that bothers me a lot with Hex is visualizations. Their features are great - I love the context studio, the app, threads, I trust their agent, and despite what some people say, I also appreciate that you have to approve and confirm agent changes to unblock it. But one thing keeps bothering me: the quality of AI-generated charts.

Not once have I been able to simply show a chart created by AI to anyone. It’s either painfully simple or wrong: incorrect chart type, wrong axis labels, not visually appealing including very questionable color choice), missing details, or else. I got used to finishing the charts myself with plots directly there in the notebook, but there had to be a better way.

Last week, the team announced improvements to visualizations: “The Hex agent can now build more beautiful charts, including dual axis and charts with reference lines”. I’ll be testing it more, truly excited about this.

Women in Data Science study - participants needed!

McGill University is running a study on gender in data science. If you’re a woman working in data science, please consider participating in this interview-based study. Participants will share their experiences, both good and bad, discuss some of their projects, and reflect on what they’ve learned.

If you’re interested, please email Emmanuelle Vaast at emmanuelle.vaast@mcgill.ca or reach out to her on LinkedIn. Your participation would be greatly appreciated.

In other news:

• If you missed Metabase JOIN, session recordings are here.

• RillData introduced Metrics SQL: A SQL-based semantic layer for humans and agents.

• CEO cancels BI tooling, replaces it with AI, breaks everything.

📈 New industry reports and benchmarks

• Customer messaging in 2026: Email, SMS, push, in-app from Customer.io

• 2026 State of Analytics Engineering Report from dbt.

• State of Digital Analytics 2026 from Mixpanel.

I’m very excited to publish my own benchmark report this month!

With the help of the ChartMogul team, I analyzed data from 3,974 companies and 4.78 million returned customers to better understand winbacks.

Winbacks are both underrated and extremely important. One of the biggest findings is that they are heavily front-loaded. About 45% happen within 30 days, and 66% happen within 90 days.

The data also shows that the longer customers stay churned, the less likely they are to return - and if they do return, they are more likely to come back at a lower value. So timing matters. With winback, every day counts.

You can see the winback benchmarks, understand where you stand, and learn how to approach winback strategy here: Why Winbacks Are One of the Most Important Growth Signals.

📚 Weekend Longread

Not a read, but a listen this time - Why Traditional Statistics Still Matters in the Age of AI. Prof. Rob Hyndman is one of the world’s most influential applied statisticians. He published over 200 research papers, co-authored more than 65 R packages, and wrote 5 books on time series forecasting. He is also a Fellow of both the Australian Academy of Science and the Academy of Social Sciences in Australia. I loved his interview on why rigorous statistical thinking remains important in the age of AI, and what data scientists are giving up when they abandon it.

⚙️Know your craft

• Why Customer Count Can Be a Misleading Metric

• 7 analysis skills to teach your AI client (and the prompts behind them).

• Test & Roll: Why Smaller A/B Tests Can Make More Money

• When the Long Tail Eats Your LTV Model

• Google Ads Maturity Model

• Why Most Time Series Models Fail Before They Start

• Regression should predict full distributions

• Schotter Plots in R

• I Let Claude Code Build ELT Connectors. Here’s What Happened

• Semantic Layer vs. Text-to-SQL: 2026 Benchmark Update

❤️ 10 favorite publications last month

When data experts become data philosophers

1. BI’s Second Unbundling from

2. Measurement Engineering: The Part of Data Science That Will Thrive in AI from

3. How do you make a chart? from

4. The 10x Data Team = The Markdown Team from

5. So you’ve been asked to “take over” some old data pipeline… from

6. Data agents: When enterprise analytics learns to reason from Pratish Yadava

7. How Analytics Drives Progress from

8. AI-Ready Data vs. Analytics-Ready Data from Animesh Kumar

9. The Power of Mathematical Modeling from

10. The average user does not exist from Saeideh Bakhshi

📊 Monthly Chart Drop

✈️ Upcoming events in April

• May 4-8, San Diego, CA: Enterprise Data World + Data Governance

• May 5-7, San Diego, CA: Tableau Conference

• May 6-7, Boston, MA: Data Summit 2026

• May 6-8, Stockholm: 11th Data Innovation Summit 2026

• May 11-13, London, UK: Gartner Data & Analytics Summit 2026

• May 12, San Francisco, CA: Mixpanel | MXP 2026

• May 12, Menlo Park, CA: Data Connect 2026

• May 12-14, San Francisco, CA: AI Conference (aka Data Council) 2026

• May 12-15, San Francisco Bay: SaaStr AI Annual 2026

• May 13, NY: DSS: The Future of Applied AI in Finance and Banking

• May 16, London, GB: Data Science Festival 2026

• May 18-19, San Jose, CA: AI & Big Data Expo

• May 19-21, Tokyo, Japan: Gartner Data & Analytics Summit 2026

• May 19-21, Las Vegas, NV: MAU 2026

• May 19-21, Las Vegas, NV: Informatica World 2026

• May 20, virtual: Semantic Layer Summit

• May 20-21, Dubai: 11th Data Innovation Summit MEA

• May 20-22, Malta: World Data Summit

• May 21, London, GB: Mixpanel | MXP 2026

• May 20-29, Chicago: Data Science Conference

• May 31-June 5, Bengaluru, India: SIGMOD/PODS Conference

Also, check this calendar for the full list of data and analytics events this year - The Biggest Data, Analytics, and AI Conferences of 2026.

Thanks for reading, everyone!

To bring you up to speed: A few years ago, data scientists from Google and Harvard collaborated on a study to prove why high-level business KPIs (revenue, North Star, ARPU, or retention) are poor metrics for A/B tests. They move too slowly, are too noisy, and often are not sensitive enough to capture the true impact of a product change in a short experiment. Instead, use more sensitive metrics, and when that still is not enough, use proxies.

They introduced the concept of Pareto Optimal Proxy Metric - metrics designed to balance 2 things that matter most in experimentation - accuracy and sensitivity. In my earlier publications, I broke down why core business KPIs are usually a poor fit for A/B tests and how to choose better proxy metrics for product launches, feature rollouts, and optimization work.

Today, I take a step further and share another part of their research - how to choose a proxy when the outcome you actually care about is long-term. There is no single best proxy metric that works forever. The right proxy depends on the size of your experiment, and the best way to choose it is to learn from your past tests rather than reuse the same metric again and again. Let’s dive into it.

Why data scientists came together on this research

It is not common for companies to invest deeply in academic research. So what made data scientists from Google, Stanford, and Duke come together around A/B testing?

Because experimentation became too important to get wrong.

As more teams learned that A/B testing improves decision-making, everyone became obsessed with running tests and scaling experimentation. Also, A/B testing tools have multiplied in the last few years. And for good reason. Companies that scale experimentation get better at product thinking, faster iteration, and more data-driven decision-making.

But there is a problem.

Not every team has the budget to hire data scientists, and many experiments are still analyzed by product, growth, marketing, or operations teams that do not always have deep statistical training. We are lucky to have much better experimentation tools today - they help teams set up tests, split traffic, and read dashboards. But making sense of impact - and connecting test results back to KPIs, real business movement, and actual opportunity, is still where most tools fail. This is also where AI will fail most of the time, unless you can feed it all of your data sources, historical context, metric definitions, and business logic. And right now, most teams are still far from that.

So teams end up in a strange situation. They run 12 A/B tests in 6 months. Every winning variant shows a +10% lift in Signup-to-Paid, LTV, or some other dashboard metric. And yet the real business KPIs keep declining over the same 6 months.

So were all 12 tests wrong?

Yes - or at least the measurement was.

As someone who has lost this argument more than once, I was genuinely excited when these papers came out. I have had to explain again and again why we should not evaluate A/B tests using ARPU or LTV (good luck with that, btw - those metrics are literally sitting right there in every subscription or SaaS dashboard for you to evaluate A/B tests, which makes this so frustrating!)

I was excited not only because one of the researchers is my good friend, Lee Richardson, who has spent years studying experimentation effects at Google, but also because these papers give practitioners something much stronger than Olga’s opinion. Now, when a CEO points to the RevenueCat or Adapty dashboard and argues that the test clearly won, while you are saying the result is inconclusive, you have academic research to back you up.

Why we need proxy metrics

The painful truth about A/B testing is that the metric you care about is usually the one you cannot measure well in time.

Whether it is long-term retention, LTV, long-term satisfaction, or future visit frequency, the real business outcome is often slow to show up, noisy, delayed, or expensive to measure. So teams end up relying on proxy metrics: shorter-term signals they hope move in the same direction.

The problem is that, in many cases, not much work is done to prove that a given proxy is the right fit for measuring a specific type of impact over a specific time frame. Most proxy metrics are either fast but wrong, or directionally right but too noisy to trust in smaller tests. This paper tries to solve that tradeoff in a more practical way that teams can actually adopt and reuse.

The problem with most proxy metrics

A lot of teams treat proxy selection as a ranking problem:

• Which short-term metric correlates best with the long-term metric?

• Which one moves fastest?

• Which one has the best historical hit rate?

Good questions, but incomplete.

The authors define proxy quality in a more practical way: how well the observed treatment effect on the proxy tracks the true treatment effect on the long-term outcome in a new experiment. In plain English, if you are going to make launch decisions based on a proxy, that proxy should do 2 things well: (a) point in the same direction as the real long-term effect, and (b) stay stable enough to be measured reliably in the experiment you are running.

A proxy can fail in 2 main ways:

• It can move in a different direction from the long-term metric, even if it looks fine in the short term.

• It can be so noisy that even a useful signal becomes unreliable, especially in smaller tests or when using late metrics like Day 30 retention or revenue.

So here is what teams should do.

How to use the history of past tests to find the right proxy

Read more

Subscribe now

Winning a customer back is one of the clearest signals of product value, pricing fit, and customer trust. But just as important is how quickly they come back.

In ChartMogul data from 3,974 companies and 4.78 million returned customers, winbacks are heavily front-loaded: 45% happen within 30 days, and 66% happen within 90 days.

Most customers who return do not come back on a cheaper plan. In fact, only 25% return on a lower-ARR plan, while 42% come back on the same plan and 33% return on a higher-ARR plan.

Time matters: the longer customers stay churned, the less likely they are to return, and the more likely they are to come back at a lower value if they do. The point of winback is not just to recover customers, but to recover revenue while it still matters.

There is also a meaningful difference between growing and declining companies. Declining companies tend to have a narrow recovery window: customers either return quickly or not at all. Growing companies still see most returns happen early, and over time they keep winning customers back for longer after churn.

That combination makes winback one of the most useful growth signals to watch. It tells you whether customers are willing to come back, how quickly they return, how much value they still see in the product, and whether your business is preserving or losing pricing power after churn.

Why winback matters

A customer who returns after churn is a stronger signal than a customer who converts for the first time.

New conversions can be driven by top-of-funnel acquisition, discounts, timing, brand, or curiosity. A returning customer is different. They already evaluated the product once, decided to leave, and then chose to pay again. That makes re-subscription a strong signal of product stickiness, perceived value, and trust.

Winback patterns also reveal something many teams miss: customer value does not disappear in a binary way. It decays over time. Some customers leave and remain highly recoverable for a short period. Others drift further away, become more price-sensitive, or return only if they can do so at a lower commitment.

Winback is a strong PMF signal

Winback deserves to be treated as more than a lifecycle campaign metric. It is also a product market fit signal. Winback patterns can also help reveal seasonality in demand and identify when customers are most likely to return.

Companies over $30M ARR show a median winback rate that is almost 2x that of companies under $500k ARR.

Winback rate seems to improve with scale. Smaller companies struggle the most with getting customers back - Under $500k ARR has the lowest median of 7%. Bigger companies may have stronger brand recognition, better lifecycle marketing, more plan/options/pricing flexibility, and more product breadth, which gives former customers more reasons to return.

The data behind this analysis

For this analysis, I used returned-customer data from ChartMogul across 3,974 organizations and 4,783,216 returned customers. The original dataset included 8,174 organizations, but I filtered out companies with missing or incomplete data to create the final sample.

The sample spans companies of different types, sizes, and ARR trajectories with very different winback rates:

About 75% of organizations in the sample have winback rates below 15%, with the largest share of companies clustered under 7%. Only 9% of companies achieve winback rates above 25%, suggesting that strong winback performance is possible, but uncommon.

To better understand how winback behavior differs by business context, I segmented organizations into 3 groups based on ARR movement: growing, stable, and declining.

This matters because company trajectory shapes winback behavior. It affects both the size of the recoverable customer pool and the conditions under which customers return.

As expected, companies with growing ARR have far more winback customers:

Growing ARR companies make up just 16% of the sample, but they account for 61% of all winback customers and post the highest winback rate at 13%. Stable ARR companies represent 74% of the sample but account for only 36% of winback customers. This suggests a clear positive relationship between winback and ARR growth.

That relationship should be interpreted carefully. Larger companies typically have more customers overall, which gives them a bigger pool of churned customers to reactivate. So, winback alone does not explain growth. Still, the pattern suggests that winback can be a meaningful growth lever, especially for companies operating at scale.

Most churned customers return early or not at all

Among customers who do return, most come back quickly. The longer a customer has been churned, the lower the chance they will return:

This chart shows how returning customers are distributed by the time between churn and re-subscription, from those who return within the first week to those who come back after more than 2 years.

Nearly half of all winbacks (45%) happen within the first 30 days, with the 8–30 day window alone accounting for 26% of all returns. This makes the first month after churn the highest-leverage window for re-engagement.

After 90 days, winbacks become much less common. In total, 66% of returning customers come back within the first 90 days.

Long-gap winbacks do happen, but they are rare. Fewer than 10% of customers return after a year, and less than 4% after two years. The median time to return is just 38 days.

The takeaway is that winbacks are heavily front-loaded. The highest-leverage window is early, especially the first month after churn. Teams that wait too long experience lower response rates and operate after the best recovery window has already passed.

This also changes how winback should be approached. It should not start only after a customer has been gone for months. In many cases, the right time to intervene is even earlier: when engagement drops below normal patterns, or when a customer cancels but still has time left in the billing period.

Growing and declining companies show different winback behavior

Companies with growing ARR have a significantly higher winback rate. They also may skew the entire distribution towards a short winback timeline. Let’s see if there is a difference in the comeback window time between our ARR groups:

And there is!

The chart above shows when churned customers come back, split by whether their company’s ARR has been growing, stable, or declining.

The “act fast” pattern holds for all 3 segments - returns are front-loaded in the first 90 days regardless of company ARR trajectory. Declining ARR companies have the sharpest early spike. Nearly ⅓ of their winbacks happen in the 8-30d window. This suggests their customers either come back quickly or not at all.

Growing ARR companies have a flatter, more spread-out curve. Their returns are more evenly distributed across time buckets (17% within the first week, 23% at 8-30 days, 23% at 31-90 days). These customers are more likely to be “recoverable” at later stages.

That matters because it changes what a winback strategy should look like. Winback campaigns for declining-ARR companies should be aggressive and immediate. For growing companies, the window is still front-loaded, but there is more space for sustained re-engagement over the following months. These companies appear better positioned to recover customers later, which may reflect stronger product value, broader use cases, better plan coverage, or stronger lifecycle execution.

This does not mean one company should act quickly and the other should not. Both should. The difference is that growth companies appear to retain customer loyalty for longer.

Most customers do not come back cheaper

A common assumption is that churned customers return only when offered a lower price, a discount, or a cheaper plan. But the data does not support that:

• 25% winback customers came back on a lower ARR plan

• 42% came back on the same plan

• 33% came back on a higher ARR

The average ARR change between the churned plan and the winback plan is +$7.77, but the median change is $0.

That is an important correction to how many teams think about winback.

Plan-term changes at winback are the exception:

Monthly → Monthly dominates all transitions, meaning most returning customers keep their original billing cadence.

The biggest shift is Monthly → Annual. That transition happens at nearly twice the rate of Annual → Monthly, suggesting that returning customers are more likely to move to a higher-commitment billing cadence than to a lower one.

The longer you wait, the lower the return rate and the lower the return value.

The longer a customer remains churned, the less likely they are to come back to the higher-tier or more expensive plan.

The chart below shows the percentage of returning customers who came back on a lower-ARR plan, broken down by how long they were churned before their winback:

Customers who return within 8–30 days have the lowest downgrade rate at 23%. For those who return after more than a year, the downgrade rate rises to 29%.

That means time away hurts revenue recoverability. Not only do fewer customers come back later, but those who do are more likely to come back at a lower spend. By the time the customer returns, part of the original pricing power has already been lost.

There is one exception - the 0-7 day bucket is slightly noisier, with a downgrade rate of 25%, likely because some of those cases reflect involuntary churn or billing issues rather than true product-driven churn. But the broader pattern still holds: the longer the gap, the weaker the revenue return.

The revenue decay is strongest in growing companies

Growing ARR companies show the biggest value decay over time. The chart below shows the percentage of returning customers who came back on a cheaper plan, by how long they were churned, and their company’s ARR segment:

Growing ARR companies have the most to lose from delayed winbacks. Their downgrade rate rises from 16% among customers who return within the first week to 28% among those who return after more than 2 years. In other words, fast winbacks at these companies do more to protect revenue per customer.

Stable ARR companies show the highest downgrade rates across most time buckets, peaking around 29%. That may point to these companies serving more price-sensitive customers, who may use churn as an opportunity to reset onto a lower-priced plan.

Declining ARR companies look flatter by comparison, with downgrade rates hovering between 21% and 25% regardless of churn duration. This may suggest that time away has less impact on return spend for these customers, possibly because many were already on lower-priced plans before churning.

Delayed winbacks tend to come back at lower value across all ARR groups, but the effect is strongest among growing companies. At those companies, customers who return within the first week keep their original plan level 84% of the time; after two years, that falls to 72%.

Delaying winbacks reduces the chance of return and the value of every return.

What teams should take away from this

1. Timing matters most. Most winbacks happen early: 45% within 30 days and 66% within 90 days. Wait too long, and you miss the highest-leverage recovery window.

2. Most returning customers are not looking for a cheaper plan. Winback is less about discounting and more about restoring relevance before value decays.

3. Delay reduces both return rate and value. The longer it takes to win a customer back, the less likely they are to return, and the more likely they are to come back at a lower ARR.

4. Recovery windows differ by company trajectory. Declining companies tend to have a narrower winback window, while growing companies sustain recoverability for longer. Winback strategy should reflect that.

5. Winback is a signal of growth quality. It shows not just whether customers return, but how quickly they return and how much value they still see when they do.

Final thought

Winback is about bringing customers back - and bringing them back fast before too much value is lost. Most winbacks happen early. As time passes, both the probability of return and the value of that return decline. That makes winback a signal of product value, pricing power, and customer trust. When a customer returns, they are choosing to pay again after already trying the product and deciding to leave. The strongest winback strategies recover customers early enough to protect both return rates and revenue.

Thank you to the ChartMogul team for sharing their data. Make sure to check their Insights page for so much more data-backed SaaS research.

Subscribe now

I’m continuing my AI series, and today I’m taking a closer look at Claude for Excel.

A quick step back: for Excel users, the AI journey started in 2024, when Microsoft launched Copilot in Excel. A year later, it also added the COPILOT() function. Neither product has been especially strong, and neither received a particularly warm reception:

Then Anthropic launched Claude in Excel, and the reaction quickly split into two camps: people who are obsessed with it, and people who aren’t impressed at all.

After using it for the past 2 months, I land somewhere in the middle, though slightly closer to the skeptical side. A lot depends on the user, the environment, and the specific use case. Claude in Excel is not something I would trust blindly with a financial model, a KPI scorecard, or any analysis where logic depends on definitions. It is still too willing to overcomplicate formulas, too inconsistent on multi-step requests, and too capable of making mistakes.

Below, I’ll walk through Claude for Excel: how to get started, where it works best, how to use it well, and where it still falls short.

Why Claude for Excel may be the most underrated AI tool in finance

It’s truly powerful - considering it’s still “officially” in beta. It’s an add-in that integrates Claude into your Excel workflow. It supports conversational format, debugging errors, root cause analysis, building new models, navigating complex multi-tab workbooks, and works with connectors to bring context from your other tools. It’s available only on Microsoft Excel - it’s not supported in Google spreadsheets.

I’m skeptical of the output accuracy and precision, but it’s a truly powerful tool in your toolbox:

Read more

It’s good timing, too - Amplitude has announced its 2026 AI Impact Awards. But I have wanted to break down Amplitude AI for a while now.

For me, Amplitude AI is helpful about 40% of the time I use it. Only about 10% of the time does it work really well - saving me time or surfacing something I had not thought of before. And that is with a good event setup designed to make analytics actionable. For most teams, I assume it is probably even less useful.

Still, imperfect AI support inside Amplitude is better than having no AI support at all.

Today, I want to break down Amplitude AI and share the workflows I use - both to show where it helps and to give you ideas for your own work. I’ll share 20 proven prompts that will save you time and will help you get more done.

Getting started with Amplitude AI

Amplitude MCP

Amplitude now has several AI products, from simple tools you can use right away to more advanced ones that require integrations.

If I’m not mistaken, Amplitude was one of the first digital analytics tools to launch an MCP server. It brings your product data and context into AI tools like Claude, Cursor, or any other MCP client. This lets you query dashboards, pull insights, and take action inside your existing workflows.

By default, if you have an Amplitude account, you likely already have access to MCP. You usually do not need to upgrade or do anything extra, unless your admins disabled it for your organization (and there may be valid reasons for that).

I’ll do a deeper dive on MCP later, so I do not derail the main topic, but if you want to explore it now, here are the steps to connect it.

Global agent

Have you noticed the floating action button in the lower-right corner of the screen? That is not just a chatbot (anymore). It is Amplitude’s Global Agent (but used to be a chatbot).

As Amplitude describes it, it helps you understand your product data in context, generate insights and explanations, and take action by carrying out complex multi-step workflows.

You can think of it as a chatbot on steroids, but its purpose is very different.

The earlier chatbot was simple and limited. That may be why many people still ignore this feature. But this is now an agent, and it can be powerful. It is worth trying.

One downside is that it can be slow, although that depends on the task and your event volume and setup. Another thing to keep in mind: if your Amplitude account is new, give it some time. In my experience, it works best when your account already has enough reporting history, charts, and dashboards. With newer accounts, it still needs time to learn your data and context.

Specialized agent

In addition to the floating Global Agent, Amplitude has also launched several specialized agents. There are 4 right now. These are purpose-built agents that focus on one domain or workflow:

Read more