AI Fundamentals

Apply AI in your role

Career Development

Career development

Craft

Product

Engineering

Design

Marketing

Leadership fundamentals

Leadership

Founders

By Role

More

Learn proven approaches for quickly improving AI applications.  Build AI that works better than the competition, regardless of the use-case.

Cohort 5

ML Engineer with 20 years of experience

Airbnb

GitHub

DataRobot

AlixPartners

Hamel Husain

ML Engineer with 20 years of experience.

ML Systems & Applied AI Evals Researcher

Google

UC Berkeley

Stanford University

Shreya Shankar

ML Systems Researcher Making AI Evaluation Work in Practice

Cohort 4

Cohort 6

All students get:<ul><li>♾️ Unlimited access to future cohorts &amp; office hours: never worry about timing or missing new material.</li><li>🗄️ Lifetime access to all materials!</li><li>🤖 6 months of unlimited access to our new AI Eval Assistant (more info below).</li><li>🧑‍🏫 10+ hours of office hours to maximize the value of live interaction.</li><li>🏫 A Discord community with continuous access to instructors to get unstuck (even after the course!).</li></ul>---Do you catch yourself asking any of the following questions while building AI applications?1. How do I test applications when the outputs require subjective judgements?2. If I change the prompt, how do I know I'm not breaking something else?3. Where should I focus my engineering efforts? Do I need to test everything?4. What if I have no data or customers, where do I start?5. What metrics should I track? What tools should I use?6. Can I automate testing and evaluation? If so, how do I trust it?If so, this course is for you.This is a flipped classroom setting. All lectures are professionally recorded with an emphasis on live office hours and student interaction.

Stop guessing if your AI works. Build the feedback loops that make it better.

How To Collect Data For Evals

Get immediate clarity and direction with Error Analysis

Implement Effective Evaluations

Master Architecture-Specific Eval Strategies

Run Evals In Production

Ensure That Evals Lead To High ROI

Engineers and PMs who ship prompt changes and hope nothing breaks. (You'll learn to measure impact before and after every change.)

Teams still spot-checking AI outputs by hand instead of measuring systematically. (You'll learn how build automated evals you can trust.)

Leaders who don't know where their AI is failing or where to invest resources. You'll learn how to systematically find &amp; prioritize issues.

Learn how to evaluate, ship, and monitor GenAI products that perform reliably in production.

Head of AI Applied Science

Shopify

Carvana

Harvard University

Stella Liu

Cofounder, AI Evals & Analytics

System1

Uptake

Seasalt.ai

UCLA

University of Washington

Amy Chen

Cofounder at AI Evals & Analytics

Cohort 2

Cohort 3

<blockquote>Building AI is easier than ever. Building AI that is reliable, safe, and trusted is not.AI evaluations help teams measure quality, detect failures, and build guardrails around your GenAI systems that will survive in the real world.</blockquote><blockquote>🔹</blockquote><h3>Who this course is for</h3>Data scientists, QA/QE professionals, and PMs who:<ul><li>have no prior background in AI evals, or</li><li>have tried LLM-as-a-judge but want a clearer framework.</li></ul><blockquote>🔹How the course works</blockquote><ul><li>Self-paced lectures you can watch anytime</li><li>Hands-on exercises using a pre-built GenAI product</li><li>Dedicated office hours for questions and guidance</li></ul><blockquote>🔹</blockquote><h3>What makes this course different</h3><ul><li>Playbook-driven approach</li><li>Evaluation-driven development</li><li>Product-focused</li><li>Hands-on experience</li><li>Flexible learning + expert support</li></ul>🔹Curious about the ideas behind this course? You can explore related topics in the <a target="_blank" rel="noopener noreferrer nofollow" href="https://datasciencexai.substack.com/">Data Science × AI Substack</a>, where we share perspectives on evaluating real GenAI products.

Learn How to Evaluate GenAI Products Confidently in 3 Weeks

Execution-ready Testing & Monitoring Playbook

GenAI Risk Management Strategy

Hands-On AI Evals Exercise

Product LeadersBuilding GenAI products and need a clear playbook for evaluation frameworks, success metrics, and shipping with confidence.

Data ScientistsLeveling up with AI product skills: learn AI evals, LLM-as-a-judge, and design metrics for GenAI systems.

QA and Test EngineerAdapting traditional testing practices to GenAI systems.

Learn practical, hands-on methods to evaluate, measure, and improve AI products using data science to make better product decisions.

Principal Data Scientist, AI Evaluations

Stripe

Nextdoor

PwC India

Ontra

AppFolio

Shane Butler

Principal Data Scientist, AI Evaluations at Ontra

Cohort 1

With the recent leaps in model capabilities, it is becoming clear that much of the AI evaluation work previously requiring humans to build and run can be automated through agentic systems. This course teaches you how to build those systems, and where human judgment and input is still required.The foundational work of designing eval frameworks, defining quality signals, and structuring failure analysis remains critical. But the execution of that work, running evaluations, scoring results, iterating on improvements, and surfacing what matters, can now be handled by agentic workflows that operate continuously and at scale.This course equips you with the capability to build, test, and deploy automated evaluation systems using Claude Code. You will leave with working systems, not just conceptual understanding.The focus is on automation that is trustworthy. Systems that score their own output, flag uncertainty, and know when to surface decisions to a human rather than proceeding autonomously.The foundational content on AI evals for product development is available free at <a target="_blank" rel="noopener noreferrer nofollow" href="https://aianalystlab.ai">aianalystlab.ai</a>. If you are new to AI evals, start there.

Build the agentic systems that automate your AI evaluation workflow from end to 

Build agentic evaluation workflows in Claude Code

Automate scoring and quality assessment

Implement iterative improvement loops

Scale experiment execution

Integrate with production data and decision-making

You will get the most out of it if you are actively working on, or about to ship, AI features in a real product.

Real-World AI Product Experience

Product Managers: Want automated eval systems for your team? Learn what is possible, reliable, and where human oversight stays essential.<pre><code class="language-markdown"></code></pre>

Data Scientists &amp; Analysts: Spending more time running evals than learning? Build systems that handle execution so you focus on decisions.

Engineers and ML Engineers: val bottlenecks slowing your release cycle? Automate repetitive work and ship with speed and confidence.

Find and prioritize your AI's biggest failures in 2 days — using your own data.

ML Engineer with 20+ years experience

Trained 4,000+ AI practitioners from 500+ companies on AI evaluation.

ML Systems Researcher 

Cohort 1: Debug Your Data

This isn't a course — we roll up our sleeves and work on your product with your data.You bring your real traces and interaction logs. Over 2 days, we systematically find where your AI is breaking down, why users are losing trust, and what to fix first. Most teams building AI products are stuck in the same loop: ship a change, hope it helps, manually spot-check a few outputs, repeat. Meanwhile, users hit failures nobody on the team has even seen.In this private workshop, we break that cycle. You bring real traces and interaction logs from your product. Together, we systematically uncover where your AI is breaking down, build a catalog of failure modes specific to your use case, and leave with a prioritized plan to fix what matters most.This methodology has been refined across 4,000+ practitioners from 500+ companies, and is part of what we teach in our <a target="_blank" rel="noopener noreferrer nofollow" href="https://maven.com/parlance-labs/evals">full-length course</a>. It's the highest-ROI activity in AI product development.Each workshop can accommodate a team of up to 6 participants. We can help you identify which team members to bring (trust us, you don't want more than 6).Below is a sample agenda, this can be customized to your needs and schedule.

Walk away knowing exactly where your AI product is failing, why, and what to fix first — with a concrete plan your team can execute immediately.

Find hidden failure patterns in your AI product

Build a prioritized roadmap to fix what matters

Design metrics that prove your fixes work

This workshop is hands-on with your data. You need a deployed product generating real interactions we can analyze together.

A live AI product or feature with real users

We work directly with your production data. You'll receive a prep guide with instructions on what to collect and how to format it.

Access to recent traces or interaction logs

Engineers, PMs, and domain experts who know the product. The best results come from cross-functional teams who can act on findings immediately.

A dedicated team of 2-5 participants

Engineering teams with a live AI product that has real users.

Product managers responsible for AI features who need a data-driven improvement plan instead of guessing which changes will move the needle.

Technical leaders who want to stop relying on manual QA and spot-checks, and instead build a systematic approach to improving AI quality.

Evals aren't your product moat. A continuous improvement flywheel is.

AI Founder & Advisor to F500s | Ex-AWS

Amazon Web Services

Microsoft

University of Oxford

Aishwarya Naresh Reganti

June Cohort

Check out our free AI Evals Certification <a target="_blank" rel="noopener noreferrer nofollow" href="https://github.com/aishwaryanr/awesome-generative-ai-guide/blob/main/free_courses/ai_evals_for_everyone/README.md">here</a> (5000+) Learners!Most teams building AI products today know about evals and LLM judges, but they’re deeply confused about how to actually turn them into actionable improvement signals post production. Just throwing in LLM judges won’t magically fix your product. It won’t.<ul><li>This course shows how to use offline evals, online evals, and production monitoring to drive continuous improvement, based on real production experience.</li><li>We believe that going forward, a thoughtfully designed data flywheel will be the moat for AI applications. This course shows you how to build it.</li><li>This course is for people who’ve already built AI systems and are stuck with evals that are noisy, expensive, or uselessIf you need something more foundational, check out our top-rated flagship<a target="_blank" rel="noopener noreferrer nofollow" href="https://maven.com/aishwarya-kiriti/genai-system-design"> enterprise AI course</a> taken by over 1500+ builders and leaders.</li><li>For questions or bulk enrollments please email problemfirst.ai@gmail.com</li><li>We follow a flipped-classroom format with async lecture content and meet twice a week for office hour style sessions to optimize two-way/interactive live time.</li></ul>

Move beyond one-time evals to continuous improvement flywheels for AI products

Understand what evals actually mean and why it’s misunderstood

Build a pre-deployment evaluation foundation from scratch

Choose the right types of metrics for the system you’re building

Monitor in production and close the improvement loop

You should have built small AI applications before and be familiar with concepts like RAG, MCP, and tool use.

A good understanding of AI system design

The course includes optional Python assignments, you're free to use coding agents, but basic coding skills are recommended

Working knowledge of Python/Coding (Optional)

Software/AI Engineers, Strategists, Data Professionals, Solution Architects and Consultants looking for systematic evaluation practices

Business Leaders and Product Managers seeking to gain the technical understanding of AI Product Lifecycle and intentional iteration

Founders &amp; teams stuck at systematic evaluations and monitoring, unsure how to turn signals into real product improvements.

From Experiments to Scale: How Leaders Build High-Performing, Trusted AI Systems

Fractional Chief Product and AI Officer

Fanatics

eBay Inc.

Walmart

Upwork

Subha Shetty

AI-first product leader, transformation coach, and advisor to F100 and start-ups

May 2026

Evals are having a moment - and for good reason. As AI systems move from pilots to production, leaders are confronting a hard truth: performance without trust doesn’t scale and AI drift is real.But evals alone are not the system.This workshop reframes evals within a broader, executive lens: performance + governance = trust at scale. You’ll learn how leading organizations are moving beyond isolated testing toward building AI systems that are measurable, reliable, and continuously improving.Grounded in insights from leading institutions and practitioners such as the WEF, McKinsey, Deloitte, Stanford HAI, and voices such as Dario Amodei, Andrew Ng and more - this session connects cutting-edge thinking to practical execution. This is a highly interactive session. You will:<ul><li>​Work hands-on to design and build your own eval (for an AI or agent use case)</li><li>​Explore how Guardian Agents, KYA (Know Your Agent), and feedback loops enable trust at scale Why This Matters Now The conversation is shifting from What can AI do? to Can this system scale and be trusted? Winning leaders will go beyond the best pilots - they will build high-performance governed AI systems that are continuously improving.</li></ul>

You will walk away with your first (or improved) eval. 
A trust framework with evals, governance, HITL, Guardian Agents and more...

Senior product, eng, data, design and business leaders responsible for:scaling AI beyond pilots and ensuring trust, safety, and governance.

AI Fundamentals

Apply AI in your role

Career Development

AI Fundamentals

Career development

Craft

AI Fundamentals

Career development

Craft

AI Fundamentals

Career development

Craft

AI Fundamentals

Craft

AI Fundamentals

Leadership fundamentals

AI Fundamentals

Leadership fundamentals

By Role

AI Evals

Cohort-based courses

You may be interested in