How to Measure AI Model Performance and Product Impact - Issue 327
My lessons for testing AI models, tracking execution, and measuring product and business impact.
Welcome to the Data Analysis Journal, a weekly newsletter about data science and analytics.
A few months ago, I wrote about how to approach product analytics when your users become AI agents or when you offer AI products: When AI Agents Become Users: Rethinking Analytics Tracking.
My main message was that traditional product analytics still works, but it no longer tells the full story. We can track whether a user completed onboarding, activated, or converted. But between onboarding and conversion, an AI product may make many decisions based on the user’s behavior - how to personalize the experience, which recommendation to provide, or how to respond.
These decisions aren’t reflected in a typical product analytics report: which model was used, which prompt shaped the response, what the agent remembered, and why the system selected that particular configuration.
So I proposed a new tracking framework to make this layer visible with a set of configuration, exposure, and outcome events.
That may sound simple and straightforward, but once you implement it, it opens Pandora’s box:
What should count as an exposure if the user never experiences the selected AI model?
What should we measure when the intended setup fails and the system switches to a different model or configuration?
Should we evaluate the experiment by user, session, task, or request? What if the AI setup can change throughout the experience? (This is my biggest challenge now to figure out.)
Did personalization improve the results, or did it just reach users who were already more likely to succeed?
Is a cheaper model still cheaper if users need more attempts (or more time) to complete the same task?
So, this is what Part 2 is about. It is relevant to product analysts and data scientists responsible for setting up analytics, event tracking, measuring AI features and quantifying their impact on business outcomes.
When setting up tracking, my rule has always been that less is more: there is no need to overwhelm the analytics system with noisy events that provide little or no value.
However, with AI, we are still figuring out what works, what doesn’t, and, more importantly - what success looks like. To be safe, I am leaning toward capturing more rather than less.
For my analysis, I need visibility into the model setup, the input provided, the configuration selected, and what was executed. I need request-level granularity so I can measure and report on what the system intended to do, what it actually delivered, and whether it moved any meaningful product or business metric.
The decision layer is becoming part of the product
I call it the “decision layer”, but I’m also seeing people call it router or orchestration layer. A router selects a path or configuration. An orchestration layer manages the workflow. I use the “decision layer” as a broader term for the logic that decides which model, prompt, tools, context, and validation a request should receive.
Basically, it is the layer that sits between the user experience, the AI configuration, and the analytics system. It reads live signals about the user and the request, then decides which model, prompt, tools, and context the agent should receive.
Why we need it: Most AI products do not process every request using the same fixed configuration. Depending on the question, the decision layer may:
Use a fast, lower-cost model to retrieve a governed metric from the semantic layer.
Use a more sophisticated model with access to SQL, experiment assignments, tracking releases, and data-quality tests.
Return a cached answer when the underlying data has not changed.
Query live warehouse data when the answer needs to be current.
Use a specialized tool or model for forecasting, anomaly detection, or customer segmentation.
Run additional validation.
Reconcile revenue against an internal billing system before returning the result.
Switch to another model or tool if the original configuration fails, times out, or produces a low-confidence answer.
To the user, these may all look like the same AI assistant or product. However, underneath, the product may follow a very different path for each request. Those decisions affect the answer’s accuracy, cost, latency, and ultimately the user’s behavior. So your team needs visibility into those decisions and the ability to quickly read and iterate them.
BTW, this is exactly the reason why feeding AI more context isn’t always a good thing. You need to be intentional and effective with it.
