Amplitude AI Analytics

Agent Evals 101: Best Practices for AI Product Teams

Amplitude

Included Health

Optimizely

Oracle

Vinay Goel

Staff AI Engineer at Amplitude

Ken Kutyn

Solutions Engineer at Amplitude with 10+ years in experimentation & AI evals.

Skip this and you'll scale problems, not performance. Here's what breaks and how to spot it early.

Why agent evals can't wait

Walk away with a practical structure you can apply to agent-driven workflows this week.

A starting eval framework for AI product teams

See how teams evolve from basic checks to a systematic, scalable measurement process.

A 3-6 month maturity path for your eval practice

Learn how to evaluate, ship, and monitor GenAI products that perform reliably in production.

Cohort 4

Head of AI Applied Science

Shopify

Carvana

Harvard University

Stella Liu

Cofounder, AI Evals & Analytics

System1

Uptake

Seasalt.ai

UCLA

University of Washington

Amy Chen

Cofounder at AI Evals & Analytics

Cohort 3

<blockquote>Building AI is easier than ever. Building AI that is reliable, safe, and trusted is not.AI evaluations help teams measure quality, detect failures, and build guardrails around your GenAI systems that will survive in the real world.</blockquote><blockquote>🔹</blockquote><h3>Who this course is for</h3>Data scientists, PMs and QA/QE professionals who:<ul><li>have no prior background in AI evals, or</li><li>have tried LLM-as-a-judge but want a clearer framework.</li></ul><blockquote>🔹How the course works</blockquote><ul><li>Self-paced lectures you can watch anytime</li><li>Hands-on exercises using a pre-built GenAI product</li><li>Dedicated office hours for questions and guidance</li></ul><blockquote>🔹</blockquote><h3>What makes this course different</h3><ul><li>Playbook-driven approach</li><li>Evaluation-driven development</li><li>Product-focused</li><li>Hands-on experience</li><li>Flexible learning + expert support</li></ul>🔹Curious about the ideas behind this course?You can explore related topics in the <a target="_blank" rel="noopener noreferrer nofollow" href="https://datasciencexai.substack.com/">Data Science × AI Substack</a>, where we share perspectives on evaluating real GenAI products.🔹💰 Purchasing Power Parity pricing adjustments upon request. ✉️ Email us: StellaxAmy@gmail.com

Learn How to Evaluate GenAI Products Confidently in 3 Weeks

Execution-ready Testing & Monitoring Playbook

GenAI Risk Management Strategy

Hands-On AI Evals Exercise

Product LeadersBuilding GenAI products and need a clear playbook for evaluation frameworks, success metrics, and shipping with confidence.

Data ScientistsLeveling up with AI product skills: learn AI evals, LLM-as-a-judge, and design metrics for GenAI systems.

QA and Test EngineerAdapting traditional testing practices to GenAI systems.

Acquire and develop a critical skill for product managers who are leading and contributing to AI products.

GenAI Product Builder @ Google, ex-Meta

Marily Nika, Ph.D AI/ML

Product Manager AI&ML @ Google

Diego Granados

Senior Data, ML and GenAI Scientist

George Zoto

Senior NLP/AI Scientist

Cohort 5

“Does it work?”... “Is it good enough?”... “Can we ship it?”...How do you answer these questions for AI products? You’re responsible for “running evals” but what does that mean?How do you choose the right metrics, interpret fuzzy results, and make a confident decision?This course gives you a framework to do just that.<ul><li>Map user value to evaluation (eval) objectives so your metrics aren’t abstract. Define success then translate it into measurable criteria.</li><li>Choose metrics you can actually maintain: capability, safety, UX friction, latency, cost and “does this reduce support tickets or increase activation.”</li><li>Set ship/no-ship thresholds you can defend to leadership.</li><li>Build lightweight workflows that work in real teams: human review where it matters, automation where it lasts, documentation that drives decisions.</li><li>Consider domain constraints (e.g., healthcare safety) and know what to avoid: silent failures, misleading proxy metrics and tests that don’t reflect production.</li><li>Tie everything to ROI: impact vs unit cost, eval coverage vs reliability, and the minimum viable monitoring you need post-launch.</li></ul>Experience AI evals through a case-based approach with a real AI product that we evaluate together.

Eliminate uncertainty in shipping AI features

Make confident ship or hold decisions for AI features

Translate user value into clear evaluation goals

Choose the right metrics for capability, safety, UX, and cost

Set defensible thresholds leadership will trust

Build lightweight evaluation workflows teams can maintain

Navigate domain constraints and avoid common failures

PMs leading AI features, growth, or platform initiatives

PMs who partner with ML teams and want to set evaluation standards

PMs who need to make clear “ship or hold” calls without doing the engineering

Build a full AI evals suite in 3 hours on a real use case

Founder @Lovelaice | 10 years in Product

Finastra

Trading Central

Printify

Madalina Turlea

Product leader with 10+ years in platform and SaaS products | Founder Lovelaice

Founder @ Lovelaice | 2x founder & CTO

Teleclinic

Nilo

SIXT Group

Freeletics

Catalina Turlea

2x founder & CTO | 14+ years in tech | Founder @Lovelaice

April

<blockquote>Whether you're building your first AI feature or exploring what AI could do in your product, there's a moment every PM hits: you look at the AI output and think "this seems okay?" but you don't have a structured way to know for sure.You've heard about evals. You know vibe checking isn't enough. But where do you actually start?This workshop takes you from that starting point to a working evaluation suit, built on a real AI feature use case, in 3 hours.You'll write a production system prompt, run it across multiple LLMs, read and annotate real AI responses to understand where they break, find failure patterns through error analysis, define automatic evals based on what you found, and measure whether your improvements actually worked.You'll leave with hands on experience on building your own evals suite, failure taxonomy, and a repeatable process to evaluate and improve any AI feature from here.Every participant works hands-on annotating AI responses, writing evals and running improvement iterations. We've run 1,500+ AI experiments with product teams and this workshop is the compressed version of what we've seen work.</blockquote>

Design, build, validate and iterate on an AI feature, hands-on

Understand the most important components of building AI features at scale

Build an evaluation suite to scale your testing

Learn how to identify failure patterns. Then iterate on both prompt and evals

The PM who wants hands-on experience on how to build high performing AI features

The PM who is overwhelmed by the technical tools and process around building AI

The product lead who wants improve the iteration speed or wants to scale manual testing with automatic evals

From Experiments to Scale: How Leaders Build High-Performing, Trusted AI Systems

Fractional Chief Product and AI Officer

Fanatics

eBay Inc.

Walmart

Upwork

Subha Shetty

AI-first product leader, transformation coach, and advisor to F100 and start-ups

June 2026

Evals are having a moment - and for good reason. As AI systems move from pilots to production, leaders are confronting a hard truth: performance without trust doesn’t scale and AI drift is real.But evals alone are not the system.This workshop reframes evals within a broader, executive lens: performance + governance = trust at scale. You’ll learn how leading organizations are moving beyond isolated testing toward building AI systems that are measurable, reliable, and continuously improving.Grounded in insights from leading institutions and practitioners such as the WEF, McKinsey, Deloitte, Stanford HAI, and voices such as Dario Amodei, Andrew Ng and more - this session connects cutting-edge thinking to practical execution. This is a highly interactive session. You will:<ul><li>​Work hands-on to design and build your own eval (for an AI or agent use case)</li><li>​Explore how Guardian Agents, KYA (Know Your Agent), and feedback loops enable trust at scale Why This Matters Now The conversation is shifting from What can AI do? to Can this system scale and be trusted? Winning leaders will go beyond the best pilots - they will build high-performance governed AI systems that are continuously improving.</li></ul>

You will walk away with your first (or improved) eval. 
A trust framework with evals, governance, HITL, Guardian Agents and more...

Diagnose AI trust issues more precisely  

Build your first eval or improve an eval 

Map 20+ trust techniques to solving the issue  

Write your Governance policy 

For product/AI leaders scaling systems<ul><li>You run evals but unsure they're measuring the right things</li><li>You need a system to catch AI drift early</li></ul>

For leaders dealing with org trust<ul><li>Stakeholders are skeptical of AI and you can't articulate why</li><li>You're asked to "make AI more reliable" </li></ul>

For leaders moving from pilots to production<ul><li>You're responsible for AI quality and inherited a system with fuzzy measurement infrastructure</li></ul>