Instrumented from zero: building the analytics layer for a B2B AI product

The team was building a B2B AI product for YouTube content creators: a RAG-based tool that indexed video transcripts and let creators query their own content library to surface clips, timestamps, and associated products. It was live in beta with ~5K users.

Amplitude was wired but not instrumented. Logging into the project, I expected to see activity but found none. Digging deeper, there were no events, no funnels, no dashboards, no users or user properties. The data layer was completely empty.

That’s when I took the initiative to instrument product analytics with Amplitude, end to end.

In an established product org, the measurement layer is already defined. Events are configured, naming conventions are set, user identity logic is already in place.

At a small AI startup, that baseline often doesn’t exist. In our case, it didn’t. We were flying blind. We didn’t know if our 5K beta users were actually using the product, where they dropped off, or whether they ever experienced the core value.

I decided to own it end to end: the strategy, the schema, the engineering coordination, the QA, and the rollout.

This was April 2025. AI coding tools weren’t yet capable enough to offload this kind of technical research. I worked through it the way a resourceful PM does:

• Worked through Amplitude’s documentation end to end
• Watched implementation walkthroughs on YouTube
• Attended Amplitude Community’s live training sessions
• Used ChatGPT as a thinking partner to work through decisions

Not because there was no other option, but because I thought it was faster and more reliable for me to deeply understand it than to hand it off and hope.

Out of those sessions, I built a working understanding of how Amplitude works:

• Events capture discrete user actions: a click, a signup, a feature used
• Event properties add context to each event: who triggered it, from which plan, on which device
• SDKs vary by surface: browser, mobile (iOS and Android), and server-side, each with different rules for when and how events fire

The most important decision wasn’t technical. It was: what are we even trying to measure?

Tracking everything is tempting. Every tap, every screen view, every UI interaction. But tracking everything buries the signal in noise.

So I started with the business questions the product needed to answer:

• Do users experience the core value? That meant tracking when a user asks a question and gets a response from the assistant.
• Are users clicking through to surfaced content (video timestamps, associated products)?
• How often are users coming back?
• Where are users dropping off before they hit core value?
• What’s the response time for generating answers?

Once I knew which user actions would answer these questions, I designed the core events. Each one mapped back to a question we needed to answer.

This is where the work got technical. I built the schema as a spec engineering could implement against. The rule I held to: every event mapped to a question we needed answered. No ‘might be useful someday’ events.

Each event included:

• Event name with a consistent naming convention
• Trigger definition: the exact condition that fires the event
• Event properties: name, type (string, number, boolean, array), required vs. optional, plain-language description
• User properties: persistent attributes on the user identity (plan, signup date, device)

The product was conversational, so I built a separate AI-specific event layer for the actions only this kind of product had:

• Query submitted
• Response latency
• Content surfaced (video clip, product)
• Engagement with surfaced content

I drew the line between product analytics and engineering observability. Product analytics measured user behavior. Engineering owned token counts and LLM failures.

Rather than ask engineering to learn Amplitude from scratch, I sent them only the docs that mapped to the SDKs they were wiring.

I worked directly with the engineering team to wire the SDK and validate events fired correctly across web and mobile.

Once implementation was done, we ran a live QA session together. We triggered actions in the product in real time while watching Amplitude’s event stream, confirming the right events were firing with the right properties.

That session caught misfires and missing properties before they made it to production. It also meant engineering understood the schema and the intent behind it, not just the technical spec.

Within 14 days of starting, we had a working Amplitude dashboard live with beta user data flowing in.

I presented the dashboard and the schema walkthrough at the team’s weekly all-hands, so everyone understood what was being tracked, what it meant, and how to read it.

I also pinned the dashboards in our internal Slack so anyone on the team could check live data anytime.

The framework was simple: start with questions, not events. Define the schema as a spec. Own the QA. That pattern became the way I approached every subsequent product in the studio, including Hash and Loft.

Each product had its own north star and its own measurement questions. But the approach to schema design, naming conventions, and dashboard structure stayed consistent across all three. The tools varied (Amplitude, PostHog, and Mixpanel, picked for different strengths around funnel analysis, session replay, and cohort tracking), but the thinking was the same.

Each rollout repeated the same playbook: training the team on the schema, walking through dashboards at all-hands, and making the data accessible to anyone who wanted to check it.

Three instrumented products. The outcome of one system.