Welcome to the Data Analysis Journal, a weekly newsletter about data science and analytics.
A quick note: I have a 20% discount for my readers for the upcoming dbt Summit in Las Vegas on Sep 15–18! If you work with dbt, analytics, or ETL, this is a must-attend event for anyone working with data. Expect breakout sessions, hands-on labs, and time to connect with fellow analytics engineers.
Hope to see you there! Use code Ext-OlgaB at checkout.
In early June, Anthropic published a guide - How Anthropic enables self-service data analytics with Claude, and the analytics world was never the same.
For a few weeks, the post was everywhere - in Slack conversations, internal discussions, and all over my LinkedIn feed.
I know it’s been 2 months since that article was published, and many people have probably forgotten about it and moved on. Well, I haven’t.
This was one of those rare historic moments when an AI lab officially confirmed something data teams have been saying for so long: you cannot simply point a model at a warehouse and expect reliable answers.
It admitted that out of the box Claude never achieved more than 21% accuracy in answering analytical questions, even after verifying Claude had read all the materials and content available. In another test, even after providing Claude with all pre-existing analytical content - all the SQL queries, notebooks, dashboards, etc, it improved accuracy by only 1%.
So what did? Data governance, encoding analytical workflows and business context as skills and human validation. Anthropic keeps human sign-off on anything reaching leadership and tests against golden datasets.
It basically confirmed you need a village of analysts to make self-service work and keep it going.
After yet another piece of LinkedIn cringe that made very little sense, I decided to feature an Anthropic article today to (a) celebrate its unusual transparency and (b) level set expectations for where analytics stands and why so many self-service initiatives will likely fail.
Feel free to forward this to your CEO if someone still thinks your data team is slow and behind.
What Anthropic proved
Anthropic clearly said - connecting Claude directly to a data warehouse through MCP gives it access to the data, but access alone does not make its answers trusted or reliable. Claude still does not know which of several similar fields contains the approved metric, which business rules it should apply, or whether a table was retired last quarter. MCP does not teach Claude how the company defines revenue, customers, activation, or churn.
Anthropic’s data shows that typically models perform only at 21% accuracy when first introduced to a new business domain. It eventually can get improved up to 95%, but only after a senior data team organized the business context, corrected mistakes, and continuously tested the system against a known set of questions. That means, 95% is not the expected accuracy of AI analytics out of the box. It takes a village to get there.
In Anthropic’s setup, the semantic layer provides trusted definitions the AI needs before it queries the data. The problem with that is most companies do not have a semantic layer, and building one can take ages. And if you already have one, supporting, maintaining, testing, and verifying it also takes resources.
Connecting an AI model directly to the warehouse (and this is what most of us are doing now) is not enough. Reliable analytics still requires shared definitions, human review, testing, and ongoing maintenance.
I love this summary from Ayush Gupta, Genloop CEO:
Anthropic’s self-service analytics post is the clearest public case any AI lab has made that you cannot point a model at a warehouse and get reliable answers. Five thousand words in, they say it directly: pointing Claude at a warehouse “can create a false sense of precision.” The rest of the post is the infrastructure required to undo that. It is the most honest public account of what production-grade agentic analytics actually demands. It is also a near-perfect illustration of why most companies will not be able to follow the same path.
The team behind “lean” self-serve analytics
Let me start with this: the company that wrote about how it successfully implemented self-serve analytics had at least 30 full-time data scientists and analysts (plus an unknown number of data and ML vendors). And it is hiring more, clearly aiming to expand its analytics team.
That may not seem like a lot for a large company. At the time, those 30 roles represented approximately 2% of Anthropic’s employees. This puts Anthropic in the same range as Meta, Google, Grammarly, and Amazon, where data roles also represent around 2% of total headcount. All are known as best-in-class, data-driven companies that invest heavily in data and analytics across tools, culture, and decision-making.
In other words, Anthropic may describe itself as lean and efficient, but it has still made a significant bet on data and analytics - at the same relative scale as MAANG companies.
Interestingly, compare that with the share of data analytics roles at fast-growing companies: Lovable at 4%, Duolingo at 5%, HubSpot at 6%, NVIDIA at 7%, and Perplexity at 8%. It almost looks as if rapid company growth might actually be connected to… having more analysts in-house. 😈
And honestly, with 30 data scientists and analysts, you can probably build a rocket, launch it using AA batteries, and produce a dashboard proving that the landing was within an acceptable margin of error.
Jokes aside, what makes data and analytics roles different (and expensive) is that much of its value is reusable. A trusted revenue definition, customer model, or experimentation framework should work for a team of 5 and remain correct for a company of 5,000. More users and more complexity certainly create additional maintenance, governance, and support work, but the requirement does not necessarily grow one-for-one with company headcount.
If you want self-serve to operate with a tolerable level of accuracy and confidence, it does not begin with an AI assistant. It begins with significant investment in data infrastructure and data talent.
Successful self-serve starts with investment in data infrastructure
Garbage in, garbage out - only faster
In analytics and BI, you get what you pay for. Investment in data governance, data quality, metadata, lineage, and business definitions goes a long way when companies try to build self-serve analytics or integrate AI into their systems.
You can’t build real intelligence on top of incomplete 3rd party data, disconnected tools, or knowledge sitting in people’s heads. Sooner or later, it all comes back to the importance of having multiple backend services, in-house user resolution and identification, a strong data management platform - and that still requires a strong team of data experts.
To be fair, garbage in, garbage out has always been a problem. But now, with so much AI-generated content and code, garbage gets created 10x easier and 100x faster.
That is why companies should invest not only in AI tools, but more importantly - in the analytics infrastructure behind them.
Teams need systems that help them validate test setups, trace where events come from, pull a list of impacted users, review user sessions, and connect dots between different data sources to catch problems before they spread.
More on getting the analytics foundation right:
Integrations are where self-serve gets really expensive
An average small company now uses between 10 and 50 data sources. An enterprise may use more than 1,000. I didn’t believe that number either, so had to look it up.
To build reasonably mature analytics or offer reliable self-service, companies must connect these sources, resolve user identities, and map the data to consistent business definitions. This is extremely difficult. No one does it seamlessly, which is why there is an entire generation of tools (CDPs) solving with varying degrees of success.
You already know my opinion on those (How To Make A Sandwich in 587 Steps).
A typical mobile app may use 15-30 different marketing and growth tools, usually connected through SDKs, APIs, or server-side integrations. Each tool can introduce its own user identifiers, metric definitions, attribution rules, and formulas. Teams eventually have to maintain 20 versions of the same KPI, or worse - one massive Excel spreadsheet trying to reconcile all of them.
I see nothing wrong with a large spreadsheet if it serves the team well. Modern spreadsheets can be dynamic, cloud-based, directly queryable, and even provide some version control. And that worked reasonably well in the dashboard era. It is not enough in the agent era.
AI agents need governance that is systematic, executable, and available wherever the work happens. The agent needs governed access to the relevant data, shared definitions, and a reliable way to connect users, transactions, events, and sessions across systems.
That does not necessarily mean you have to copy everything into one warehouse. You can make data and business logic accessible through connectors and map them to the same identity and semantic framework. But the underlying problem remains the same for all of us: can we accurately identify the same customer across 50 different tools and connect that person to the correct transactions, events, and sessions? No AI model solves that.
Reliable and trusted analytics needs expert owners
Let the right people own the right work
I guess I belong to the annoying old-school group that still believes relevant experience matters. Probably because some of the biggest messes I have seen were created when people without the necessary expertise were asked to own and maintain particular stacks or systems.
At an early-stage company, it is normal for a tech lead to handle data + security + infrastructure + reporting. It works to a degree, and at that stage, it is often the only practical option. The problems begin when the company grows and teams continue building on top of that temporary foundation.
There should be a special form of punishment for:
Data analysts owning production data engineering.
Software engineers defining data models and business reporting.
Product managers creating “napkin math” for financial metrics and revenue forecasts.
Marketing managers interpreting A/B tests.
Data engineers or DBAs tuning ML or optimizing models.
People can and should work across functions, especially on small teams. But critical systems and projects (building a data warehouse, implementing event tracking, establishing reporting and experimentation, designing data models, and creating forecasting or financial estimates) need clear owners with the right expertise.
Many teams initially optimize for cost and speed by being “lean” and removing short-term bottlenecks. Later, they pay for that through immature analytics, unreliable reporting, and broken trust in their data:
For Airbnb, the semantic reset took almost 3 years. At Uber, adoption of the new metrics system took at least 18 months. A full transformation (migrating data, rebuilding models and BI systems, and achieving company-wide adoption) can take a few years.
Even with the right infrastructure and owners, self-serve runs into one more limitation: the tool can retrieve data, but it can’t decide what the business should be asking.
AI can’t ask the right question for you
I’m convinced that fully self-service is a myth. In my experience, it mostly fails, but not because of data or accuracy- it fails because of (a) questions asked and (b) semantics.
Stakeholders don’t ask “How many MAU we have in France” or “What’s our net new revenue on iOS?”. That would be easy to solve, and most mature companies have these stats easily available.
Instead, they ask:
“If we increase X by 2x, how much will Y improve?”
“What can I change in A, B, or C to generate an additional $200K next month?”
“If we sunset features X and Y, how much of a hit to retention should we expect?”
These questions require more than retrieving a number. They involve identifying relationships, making assumptions, estimating unknowns, and often building a model. They also require someone who knows whether the question can even be answered with the available data.
Unfortunately, Claude will still give you an answer, even when the necessary data does not exist, the assumptions are wrong, or the underlying data is unreliable. That means users must be able to question the result, challenge its assumptions, and validate the output. Not every stakeholder can (or should) be expected to do that.
This is why a large part of an analyst’s job is reframing the question. The conversation often ends with either, “We can’t answer that with the data we have” or “We can’t answer A, but we can provide B instead”.
Now imagine that the let-me-help-you-ask-the-right-question layer is gone. What happens?
I once spent more than 4 months building an internal querying tool. It allowed business teams without SQL knowledge to pull data for marketing initiatives, media profiles, OKR estimates, and whatever. The goal was to free analysts from daily, repetitive ad hoc requests.
The outcome was that teams could retrieve the data themselves, but they needed even more of the analysts’ time. They needed help understanding whether the data actually answered their question, whether the numbers were correct, and whether they were comparing apples to apples or apples to oranges. On top of that, the analytics team became responsible for access management, onboarding, user support, documentation, and ongoing tool maintenance. In short - no, it didn’t free up analysts from data requests.
As Ken Rudin, former head of User Growth and Analytics at Google and former head of Analytics at Facebook, put it more than a decade ago:
“I would rather have great people using poor technology than great systems and a team of people who don’t know how to use data. In the end, big data is only as effective and impactful as the people you have to analyze it.”
+100 to every word - and it’s even more relevant in 2026.
Thanks for reading, everyone!