AI Feature Validation for Product Managers

Whether you're building your first AI feature or exploring what AI could do in your product, there's a moment every PM hits: you look at the AI output and think "this seems okay?" but you don't have a structured way to know for sure.

You've heard about evals. You know vibe checking isn't enough. But where do you actually start?

This workshop takes you from that starting point to a working evaluation framework, built on your use case, with your data, in 3 hours.

You'll write a prompt, run it across multiple LLMs, read and annotate real AI responses to understand where they break, find failure patterns through error analysis, define automatic metrics based on what you found, and measure whether your improvements actually worked.

You'll leave with your own accuracy metric, a failure taxonomy for your use case, and a repeatable process to evaluate and improve any AI feature from here.

Every participant works hands-on on their own use case. We've run 1,000+ AI experiments with product teams and this workshop is the compressed version of what we've seen work.