Amplitude AI Analytics

Build an AI Growth Optimization System

Amplitude

Eppo

Cameo

SpotHero

PowerReviews

Eric Metelka

Product Leader, Amplitude Experiment |
Lenny's Top 25 Contributor

Recognize where evals stop giving signal and when to run a live experiment

Identify where evals end and experiments begin

Apply a practical framework for sequencing eval checks, online A/B tests, and Amplitude analytics across the AI product

Layer evals, experiments, and analytics

Use AI-native tooling to generate hypotheses, run tests, and surface analysis without manual overhead at each step.

Automate your experimentation loop

Learn proven approaches for quickly improving AI applications.  Build AI that works better than the competition, regardless of the use-case.

ML Engineer with 20 years of experience

Airbnb

GitHub

DataRobot

AlixPartners

Hamel Husain

ML Engineer with 20 years of experience.

ML Systems & Applied AI Evals Researcher

Google

UC Berkeley

Stanford University

Shreya Shankar

ML Systems Researcher Making AI Evaluation Work in Practice

Cohort 5

Cohort 6

🚨 New: September 2026 cohort's material is completely refreshed to cover advancements in the field 🚨All students get:<ul><li>♾️ Unlimited access to future cohorts &amp; office hours: never worry about timing or missing new material.</li><li>🗄️ Lifetime access to all materials!</li><li>🤖 6 months of unlimited access to our new AI Eval Assistant (more info below).</li><li>🧑‍🏫 10+ hours of office hours to maximize the value of live interaction.</li><li>🏫 A Discord community with continuous access to instructors to get unstuck (even after the course!).</li></ul>---Do you catch yourself asking any of the following questions while building AI applications?1. How do I test applications when the outputs require subjective judgements?2. If I change the prompt, how do I know I'm not breaking something else?3. Where should I focus my engineering efforts? Do I need to test everything?4. What if I have no data or customers, where do I start?5. What metrics should I track? What tools should I use?6. Can I automate testing and evaluation? If so, how do I trust it?If so, this course is for you.All sessions are live and recorded.

Stop guessing if your AI works. Build the feedback loops that make it better.

How To Collect Data For Evals

Get immediate clarity and direction with Error Analysis

Implement Effective Evaluations

Master Architecture-Specific Eval Strategies

Run Evals In Production

Ensure That Evals Lead To High ROI

Engineers and PMs who ship prompt changes and hope nothing breaks. (You'll learn to measure impact before and after every change.)

Teams still spot-checking AI outputs by hand instead of measuring systematically. (You'll learn how build automated evals you can trust.)

Leaders who don't know where their AI is failing or where to invest resources. You'll learn how to systematically find &amp; prioritize issues.

Learn how to evaluate, ship, and monitor GenAI products that perform reliably in production.

Cohort 4

Head of AI Applied Science

Shopify

Carvana

Harvard University

Stella Liu

Cofounder, AI Evals & Analytics

System1

Uptake

Seasalt.ai

UCLA

University of Washington

Amy Chen

Cofounder at AI Evals & Analytics

Cohort 3

<blockquote>Building AI is easier than ever. Building AI that is reliable, safe, and trusted is not.AI evaluations help teams measure quality, detect failures, and build guardrails around your GenAI systems that will survive in the real world.</blockquote><blockquote>🔹</blockquote><h3>Who this course is for</h3>Data scientists, PMs and QA/QE professionals who:<ul><li>have no prior background in AI evals, or</li><li>have tried LLM-as-a-judge but want a clearer framework.</li></ul><blockquote>🔹How the course works</blockquote><ul><li>Self-paced lectures you can watch anytime</li><li>Hands-on exercises using a pre-built GenAI product</li><li>Dedicated office hours for questions and guidance</li></ul><blockquote>🔹</blockquote><h3>What makes this course different</h3><ul><li>Playbook-driven approach</li><li>Evaluation-driven development</li><li>Product-focused</li><li>Hands-on experience</li><li>Flexible learning + expert support</li></ul>🔹Curious about the ideas behind this course?You can explore related topics in the <a target="_blank" rel="noopener noreferrer nofollow" href="https://datasciencexai.substack.com/">Data Science × AI Substack</a>, where we share perspectives on evaluating real GenAI products.🔹💰 Purchasing Power Parity pricing adjustments upon request. ✉️ Email us: StellaxAmy@gmail.com

Learn How to Evaluate GenAI Products Confidently in 3 Weeks

Execution-ready Testing & Monitoring Playbook

GenAI Risk Management Strategy

Hands-On AI Evals Exercise

Product LeadersBuilding GenAI products and need a clear playbook for evaluation frameworks, success metrics, and shipping with confidence.

Data ScientistsLeveling up with AI product skills: learn AI evals, LLM-as-a-judge, and design metrics for GenAI systems.

QA and Test EngineerAdapting traditional testing practices to GenAI systems.

Acquire and develop a critical skill for product managers who are leading and contributing to AI products.

GenAI Product Builder @ Google, ex-Meta

Marily Nika, Ph.D AI/ML

Product Manager AI&ML @ Google

Diego Granados

Senior Data, ML and GenAI Scientist

George Zoto

Senior NLP/AI Scientist

“Does it work?”... “Is it good enough?”... “Can we ship it?”...How do you answer these questions for AI products? You’re responsible for “running evals” but what does that mean?How do you choose the right metrics, interpret fuzzy results, and make a confident decision?This course gives you a framework to do just that.<ul><li>Map user value to evaluation (eval) objectives so your metrics aren’t abstract. Define success then translate it into measurable criteria.</li><li>Choose metrics you can actually maintain: capability, safety, UX friction, latency, cost and “does this reduce support tickets or increase activation.”</li><li>Set ship/no-ship thresholds you can defend to leadership.</li><li>Build lightweight workflows that work in real teams: human review where it matters, automation where it lasts, documentation that drives decisions.</li><li>Consider domain constraints (e.g., healthcare safety) and know what to avoid: silent failures, misleading proxy metrics and tests that don’t reflect production.</li><li>Tie everything to ROI: impact vs unit cost, eval coverage vs reliability, and the minimum viable monitoring you need post-launch.</li></ul>Experience AI evals through a case-based approach with a real AI product that we evaluate together.

Eliminate uncertainty in shipping AI features

Make confident ship or hold decisions for AI features

Translate user value into clear evaluation goals

Choose the right metrics for capability, safety, UX, and cost

Set defensible thresholds leadership will trust

Build lightweight evaluation workflows teams can maintain

Navigate domain constraints and avoid common failures

PMs leading AI features, growth, or platform initiatives

PMs who partner with ML teams and want to set evaluation standards

PMs who need to make clear “ship or hold” calls without doing the engineering

Evals aren't your product moat. A continuous improvement flywheel is.

AI Founder & Advisor to F500s | Ex-AWS

Amazon Web Services

Microsoft

University of Oxford

Aishwarya Naresh Reganti

June Cohort

Check out our free AI Evals Certification <a target="_blank" rel="noopener noreferrer nofollow" href="https://github.com/aishwaryanr/awesome-generative-ai-guide/blob/main/free_courses/ai_evals_for_everyone/README.md">here</a> (5000+) Learners!Most teams building AI products today know about evals and LLM judges, but they’re deeply confused about how to actually turn them into actionable improvement signals post production. Just throwing in LLM judges won’t magically fix your product. It won’t.<ul><li>This course shows how to use offline evals, online evals, and production monitoring to drive continuous improvement, based on real production experience.</li><li>We believe that going forward, a thoughtfully designed data flywheel will be the moat for AI applications. This course shows you how to build it.</li><li>This course is for people who’ve already built AI systems and are stuck with evals that are noisy, expensive, or uselessIf you need something more foundational, check out our top-rated flagship<a target="_blank" rel="noopener noreferrer nofollow" href="https://maven.com/aishwarya-kiriti/genai-system-design"> enterprise AI course</a> taken by over 1500+ builders and leaders.</li><li>For questions or bulk enrollments please email problemfirst.ai@gmail.com</li><li>We follow a flipped-classroom format with async lecture content and meet twice a week for office hour style sessions to optimize two-way/interactive live time.</li></ul>

Move beyond one-time evals to continuous improvement flywheels for AI products

Understand what evals actually mean and why it’s misunderstood

Build a pre-deployment evaluation foundation from scratch

Choose the right types of metrics for the system you’re building

Monitor in production and close the improvement loop

You should have built small AI applications before and be familiar with concepts like RAG, MCP, and tool use.

A good understanding of AI system design

The course includes optional Python assignments, you're free to use coding agents, but basic coding skills are recommended

Working knowledge of Python/Coding (Optional)

Software/AI Engineers, Strategists, Data Professionals, Solution Architects and Consultants looking for systematic evaluation practices

Business Leaders and Product Managers seeking to gain the technical understanding of AI Product Lifecycle and intentional iteration

Founders &amp; teams stuck at systematic evaluations and monitoring, unsure how to turn signals into real product improvements.