When AI Agents Become Users: Rethinking Analytics Tracking - Issue 316
Navigating analytics for AI agents: tracking conversations, testing prompts, and optimizing products in real time.
For the past decade, the product analytics playbook has been built around clicks, funnels, and conversion flows.
Before a product launch, teams would design tracking systems to capture key user actions. Those actions would later become metrics: signup started, trial started, checkout completed, feature used, subscription canceled.
Experimentation used the same logic. Product analysts had to decide how to instrument each test: where to place tracking, to make analytics cost-efficient and effective, how to measure the lift between control and variant, and whether to send a separate experiment_started event or simply attach the experiment group as a property on an event like trial_started.
This is the analytics world we built: instrument the user action, translate it into a metric, and measure whether the product change moved that metric.
But what happens when users become agents? Even more, what happens when users become conversational, generative, or self-evolving agents that are dynamic and learn from every interaction?
That is where traditional product analytics starts to break.
This publication is about how to navigate and adapt your product analytics setup as we transition from tracking user actions to examining agent interactions. It is about moving beyond basic event logs (which are no longer enough) and treating AI model selection as a core product experiment. That means A/B testing LLMs, prompts, voices, and agent behaviors against key metrics like retention, conversion, and LTV. Also, we’ll look at what analytics looks like for self-evolving products: systems that use real-time data to learn what works, adjust their own settings, and turn insights into action faster.
Product analytics still works. It just does not see the AI layer yet.
Product analytics is still life-essential for ensuring you aren’t getting “garbage in”.
The traffic still comes in. The events still fire. The funnel still exists. Mixpanel, Amplitude, SDK integrations, and all of that tracking are still must-haves.
But the funnel no longer explains the full outcome.
In a traditional product, the page is the product surface. A user sees a button, a form, a paywall, a checkout page, or an onboarding screen. If conversion changes, analysts can usually inspect the interface and look for a visible explanation: maybe the CTA changed, the page layout shifted, the copy was updated, the pricing page was redesigned, or the onboarding flow introduced new friction.
In an AI product, the visible interface may not change at all.
The same user may open the same screen, start the same session, and complete the same flow. But behind the scenes, the experience may be completely different.
The model may be different. The prompt may be different. The voice prompting the prompt may be different. The memory window may be different. The tools available to the agent may be different. The routing logic may be different.
From the user’s perspective, it may feel like the same product. But from the system’s perspective, it is not the same product at all - it could be 42 variations of different setups. The agent itself becomes part of the product surface.
That means agent behavior needs to be instrumented as carefully as user behavior.
The old analytics layer is not enough
In UI analytics, sessions or clicks become events.
We track button clicks, page views, signup starts, card toggles and banner hoovers, trial starts, purchase completions, and cancellations. We use those events to build funnels, cohorts, retention curves, and conversion rates.
That still matters, but in agent-based products, turns or switches become events.
Every conversation turn, model response, tool call, routing decision, voice selection, prompt variant, and memory decision can affect the user experience. This requires a 2nd layer of instrumentation to address agent analytics.
