AI Fundamentals

Apply AI in your role

Career Development

Career development

Craft

Product

Engineering

Design

Marketing

Leadership fundamentals

Leadership

Founders

By Role

More

Learn proven approaches for quickly improving AI applications.  Build AI that works better than the competition, regardless of the use-case.

Cohort 5

ML Engineer with 20 years of experience

Airbnb

GitHub

DataRobot

AlixPartners

Hamel Husain

ML Engineer with 20 years of experience.

ML Systems & Applied AI Evals Researcher

Google

UC Berkeley

Stanford University

Shreya Shankar

ML Systems Researcher Making AI Evaluation Work in Practice

Cohort 4

🚨 We are enrolling students in our March cohort until 3/30/2026. The next one won't be until September. 🚨All students get:<ul><li>♾️ Unlimited access to future cohorts &amp; office hours: never worry about timing or missing new material.</li><li>🗄️ Lifetime access to all materials!</li><li>🤖 6 months of unlimited access to our new AI Eval Assistant (more info below).</li><li>🧑‍🏫 10+ hours of office hours to maximize the value of live interaction.</li><li>🏫 A Discord community with continuous access to instructors to get unstuck (even after the course!).</li></ul>---Do you catch yourself asking any of the following questions while building AI applications?1. How do I test applications when the outputs require subjective judgements?2. If I change the prompt, how do I know I'm not breaking something else?3. Where should I focus my engineering efforts? Do I need to test everything?4. What if I have no data or customers, where do I start?5. What metrics should I track? What tools should I use?6. Can I automate testing and evaluation? If so, how do I trust it?If so, this course is for you.This is a flipped classroom setting. All lectures are professionally recorded with an emphasis on live office hours and student interaction.

Stop guessing if your AI works. Build the feedback loops that make it better.

How To Collect Data For Evals

Get immediate clarity and direction with Error Analysis

Implement Effective Evaluations

Master Architecture-Specific Eval Strategies

Run Evals In Production

Ensure That Evals Lead To High ROI

Engineers and PMs who ship prompt changes and hope nothing breaks. (You'll learn to measure impact before and after every change.)

Teams still spot-checking AI outputs by hand instead of measuring systematically. (You'll learn how build automated evals you can trust.)

Leaders who don't know where their AI is failing or where to invest resources. You'll learn how to systematically find &amp; prioritize issues.

Learn practical, hands-on methods to evaluate, measure, and improve AI products using data science to make better product decisions.

Principal Data Scientist, AI Evaluations

Stripe

Nextdoor

PwC India

Ontra

AppFolio

Shane Butler

Principal Data Scientist, AI Evaluations at Ontra

Cohort 1

With the recent leaps in model capabilities, it is becoming clear that much of the AI evaluation work previously requiring humans to build and run can be automated through agentic systems. This course teaches you how to build those systems, and where human judgment and input is still required.The foundational work of designing eval frameworks, defining quality signals, and structuring failure analysis remains critical. But the execution of that work, running evaluations, scoring results, iterating on improvements, and surfacing what matters, can now be handled by agentic workflows that operate continuously and at scale.This course equips you with the capability to build, test, and deploy automated evaluation systems using Claude Code. You will leave with working systems, not just conceptual understanding.The focus is on automation that is trustworthy. Systems that score their own output, flag uncertainty, and know when to surface decisions to a human rather than proceeding autonomously.The foundational content on AI evals for product development is available free at <a target="_blank" rel="noopener noreferrer nofollow" href="https://aianalystlab.ai">aianalystlab.ai</a>. If you are new to AI evals, start there.

Build the agentic systems that automate your AI evaluation workflow from end to 

Build agentic evaluation workflows in Claude Code

Automate scoring and quality assessment

Implement iterative improvement loops

Scale experiment execution

Integrate with production data and decision-making

You will get the most out of it if you are actively working on, or about to ship, AI features in a real product.

Real-World AI Product Experience

Product Managers: Want automated eval systems for your team? Learn what is possible, reliable, and where human oversight stays essential.<pre><code class="language-markdown"></code></pre>

Data Scientists &amp; Analysts: Spending more time running evals than learning? Build systems that handle execution so you focus on decisions.

Engineers and ML Engineers: val bottlenecks slowing your release cycle? Automate repetitive work and ship with speed and confidence.

Learn how to evaluate, ship, and monitor GenAI products that perform reliably in production.

Head of AI Applied Science

Shopify

Carvana

Harvard University

Stella Liu

Cofounder, AI Evals & Analytics

System1

Uptake

Seasalt.ai

UCLA

University of Washington

Amy Chen

Cofounder at AI Evals & Analytics

Cohort 2

Cohort 3

<blockquote>Building AI is easier than ever. Building AI that is reliable, safe, and trusted is not.AI evaluations help teams measure quality, detect failures, and build guardrails around your GenAI systems that will survive in the real world.</blockquote><blockquote>🔹</blockquote><h3>Who this course is for</h3>Data scientists, QA/QE professionals, and PMs who:<ul><li>have no prior background in AI evals, or</li><li>have tried LLM-as-a-judge but want a clearer framework.</li></ul><blockquote>🔹How the course works</blockquote><ul><li>Self-paced lectures you can watch anytime</li><li>Hands-on exercises using a pre-built GenAI product</li><li>Dedicated office hours for questions and guidance</li></ul><blockquote>🔹</blockquote><h3>What makes this course different</h3><ul><li>Playbook-driven approach</li><li>Evaluation-driven development</li><li>Product-focused</li><li>Hands-on experience</li><li>Flexible learning + expert support</li></ul>🔹Curious about the ideas behind this course? You can explore related topics in the <a target="_blank" rel="noopener noreferrer nofollow" href="https://datasciencexai.substack.com/">Data Science × AI Substack</a>, where we share perspectives on evaluating real GenAI products.

Learn How to Evaluate GenAI Products Confidently in 3 Weeks

Execution-ready Testing & Monitoring Playbook

GenAI Risk Management Strategy

Hands-On AI Evals Exercise

Product LeadersBuilding GenAI products and need a clear playbook for evaluation frameworks, success metrics, and shipping with confidence.

Data ScientistsLeveling up with AI product skills: learn AI evals, LLM-as-a-judge, and design metrics for GenAI systems.

QA and Test EngineerAdapting traditional testing practices to GenAI systems.

Evals aren't your product moat. A continuous improvement flywheel is.

AI Founder & Advisor to F500s | Ex-AWS

Amazon Web Services

Microsoft

University of Oxford

Aishwarya Naresh Reganti

June Cohort

Check out our free AI Evals Certification <a target="_blank" rel="noopener noreferrer nofollow" href="https://github.com/aishwaryanr/awesome-generative-ai-guide/blob/main/free_courses/ai_evals_for_everyone/README.md">here</a> (5000+) Learners!Most teams building AI products today know about evals and LLM judges, but they’re deeply confused about how to actually turn them into actionable improvement signals post production. Just throwing in LLM judges won’t magically fix your product. It won’t.<ul><li>This course shows how to use offline evals, online evals, and production monitoring to drive continuous improvement, based on real production experience.</li><li>We believe that going forward, a thoughtfully designed data flywheel will be the moat for AI applications. This course shows you how to build it.</li><li>This course is for people who’ve already built AI systems and are stuck with evals that are noisy, expensive, or uselessIf you need something more foundational, check out our top-rated flagship<a target="_blank" rel="noopener noreferrer nofollow" href="https://maven.com/aishwarya-kiriti/genai-system-design"> enterprise AI course</a> taken by over 1500+ builders and leaders.</li><li>For questions or bulk enrollments please email problemfirst.ai@gmail.com</li><li>We follow a flipped-classroom format with async lecture content and meet twice a week for office hour style sessions to optimize two-way/interactive live time.</li></ul>

Move beyond one-time evals to continuous improvement flywheels for AI products

Understand what evals actually mean and why it’s misunderstood

Build a pre-deployment evaluation foundation from scratch

Choose the right types of metrics for the system you’re building

Monitor in production and close the improvement loop

You should have built small AI applications before and be familiar with concepts like RAG, MCP, and tool use.

A good understanding of AI system design

The course includes optional Python assignments, you're free to use coding agents, but basic coding skills are recommended

Working knowledge of Python/Coding (Optional)

Software/AI Engineers, Strategists, Data Professionals, Solution Architects and Consultants looking for systematic evaluation practices

Business Leaders and Product Managers seeking to gain the technical understanding of AI Product Lifecycle and intentional iteration

Founders &amp; teams stuck at systematic evaluations and monitoring, unsure how to turn signals into real product improvements.

AI Fundamentals

Apply AI in your role

Career Development

AI Fundamentals

Career development

Craft

AI Fundamentals

Career development

Craft

AI Fundamentals

Career development

Craft

AI Fundamentals

Craft

AI Fundamentals

Leadership fundamentals

AI Fundamentals

Leadership fundamentals

By Role

AI Evals

Cohort-based courses

You may be interested in