Cohort 1 has $1,000 in modal credits.  This is NOT guaranteed for Cohort 2

Cohort 1

Product managers who drive strategy, development, and launch of products.

Engineers and engineering leaders who build, deploy, and scale software systems.

Measuring and evaluating AI model performance, quality, and reliability for production use cases.

Applying AI tools and techniques to professional workflows — assign alongside more specific AI topics.

Build a real AI agent, find where it breaks, and improve it with evals you can trust, working the full loop hands-on.

ML Engineer with 25 years of experience

Airbnb

GitHub

DataRobot

AlixPartners

Hamel Husain

ML Engineer with 20 years of experience.

ML Systems & Applied AI Evals Researcher

Google

UC Berkeley

Stanford University

Shreya Shankar

ML Systems Researcher Making AI Evaluation Work in Practice

AI Evals For Engineers & PMs

Learn proven approaches for quickly improving AI applications.  Build AI that works better than the competition, regardless of the use-case.

🚨 The next cohort is in January 2026. We are aiming for the next cohort to be VERY SMALL. Seats are extremely limited.🚨 All students in this course get:---- 🗄️ Lifetime access to all materials and recordings!- 🤖 6 months of unlimited access to our new AI Eval Assistant (more info towards bottom of this page).- 🧑‍🏫 8+ hours of office hours to maximize the value of live interaction. - 🏫 Lifetime Access to a Discord community with 1k+ students and instructors. This is a flipped classroom setting. All lectures are professionally edited and recorded, with an emphasis on live office hours and student interaction. Do you catch yourself asking any of the following questions while building AI applications? 1. How do I test applications when the outputs are stochastic and require subjective judgements? 2. If I change the prompt, how do I know I'm not breaking something else? 3. Where should I focus my engineering efforts? Do I need to test everything? 4. What if I have no data or customers, where do I start? 5. What metrics should I track? What tools should I use? Which models are best? 6. Can I automate testing and evaluation? If so, how do I trust it? If you aren't sure about the answers to these questions, this course is for you. This is a hands-on course for engineers and technical PMs. Ideal for those who are comfortable coding or vibe coding. ---WHAT TO EXPECT This course will provide you with hands-on experience. Get ready to sweat through exercises, code and data! We will meet two times a week for four weeks, with generous office hours (read below for course schedule). We will also hold office hours and host Discord community where you can communicate with us and each other. In return, you will be rewarded with skills that will set you apart from the competition by a wide margin. (see testimonials below). All sessions will be recorded and available to students asynchronously.--- COURSE CONTENT Lesson 1: Fundamentals &amp; Lifecycle LLM Application Evaluation- Why evaluation matters for LLM applications - business impact and risk mitigation- Challenges unique to evaluating LLM outputs - common failure modes and context-dependence- The lifecycle approach from development to production- Basic instrumentation and observability for tracking system behavior- Introduction to error analysis and methods for categorizing failures Lesson 2: Systematic Error Analysis- Bootstrap data through effective synthetic data generation- Annotation strategies and quantitative analysis of qualitative data- Translating error findings into actionable improvements- Avoiding common pitfalls in the analysis process- Practical exercise: Building and iterating on an error tracking system Lesson 3: Implementing Effective Evaluations- Defining metrics using code-based and LLM-judge approaches- Techniques for evaluating individual outputs and overall system performance- Organizing datasets with proper structure for inputs and reference data- Practical exercise: Building an automated evaluation pipeline Lesson 4: Collaborative Evaluation Practices- Designing efficient team-based evaluation workflows- Statistical methods for measuring inter-annotator agreement- Techniques for building consensus on evaluation criteria- Practical exercise: Collaborative alignment in breakout groups Lesson 5: Architecture-Specific Evaluation Strategies- Evaluating RAG systems for retrieval relevance and factual accuracy- Testing multi-step pipelines to identify error propagation- Assessing appropriate tool use and multi-turn conversation quality- Multi-modal evaluation for text, image, and audio interactions- Practical exercise: Creating targeted test suites for different architectures Lesson 6: Production Monitoring &amp; Continuous Evaluation- Implementing traces, spans, and session tracking for observability- Setting up automated evaluation gates in CI/CD pipelines- Methods for consistent comparison across experiments- Implementing safety and quality control guardrails- Practical exercise: Designing an effective monitoring dashboard Lesson 7: Efficient Continuous Human Review Systems- Strategic sampling approaches for maximizing review impact- Optimizing interface design for reviewer productivity- Practical exercise: Implementing a continuous feedback collection system Lesson 8: Cost Optimization- Quantifying value versus expenditure in LLM applications- Intelligent model routing based on query complexity- Practical exercise: Optimizing a real-world application for cost efficiency

Eliminate the guesswork of building AI applications with data-driven approaches.

Engineers &amp; Technical PMs building AI products who are interested in moving beyond proof-of-concepts.

Those interested in moving beyond vibe-checks to data driven measurements you can trust, even when outputs are stochastic or subjective.

Founders and leaders who are unsure of the failure modes of their AI applications and where to allocate resources.

Who this course is for

What you’ll get out of this course

We've tried all of them so you don't have to.

Acquire the best tools for finding, diagnosing, and prioritizing AI errors.

And how to best leverage data when you do have users.

Learn how to bootstrap with synthetic data for testing before you have users

Data flywheels ensure you have examples to draw from for prompts, tests, and fine-tuning.

Create a data flywheel for your applications that guarantees your AI will improve over time.

How can you really trust LLM-as-a-judge? How should you design them? We will show you how. We will also show you how to refine prompts, generate metadata, and other tasks with the assistance of AI.

Automate parts of your AI evaluation with approaches that allow you to actually trust and rely on them.

We will show you approaches to discover all the ways AI is not performing

Ensure your AI is aligned to your preferences, tastes and judgement.

There are an infinite number of things you can try, tests you can write, and data you can look at. We will show you a data-driven process that helps you prioritize the most important problems so you can avoid wasting time and money.

Avoid common mistakes we've seen across 35+ AI implementations.

We will provide end-to-end exercises, examples and code to make sure you come away with the skills you need. We will NOT just throw a bunch of slides at you!

Hands-On Exercises, Examples and Code

Generous office hours ensures students can ask questions about their specific issues and interests.

Personalized Instruction

Revisit the materials and lectures anytime.  Recordings and slides are made available to all students.

Lifetime Access to All Recordings & Materials

AI assistant powered by everything we've ever said about evals: course material, papers, blogs, talks, etc.  Exclusive to students only.

Unlimited Access To Our AI Evals Assistant For 6 Months

Private discord for questions, job leads, and ongoing support from the community (over 1000+ students and growing).

Lifetime Access To Discord Community

Open office hours for questions and personalized feedback.
 

8+ Office Hour Q&As

We provide a comprehensive course reader with detailed notes to serve as a future reference on evals

150+ Page Course Reader (Draft of Our O'Reilly Book)

Optional coding assignments & walkthrough videos so you can practice every concept.

 
 

4 Homework Assignments With Solutions & Walkthroughs

High production quality lectures, edited & organized by chapters (with notes) to save you time.  This is the only course on Maven like this.

Professionally Edited Lectures (Unique To This Course)

Share your new skills with your employer or on LinkedIn.

Certificate of Completion

Provide a brief description for each module as a way to sell the value of what students will be learning throughout the course.

First Module

Second Module

Third Module

Course syllabus

Instructors are recognized experts, with hands-on experience with 30+ companies

See what our students have to say

"After taking this course...I feel like a wizard who can build high quality AI agents and applications."

Can PMs also get value from this course?  Yes!  This is what PMs are saying:

6 Months Unlimited Access To An AI Evals Assistant With Access To Everything We've Ever Said About Evals

View Our Evals Hands-On Primer

Meet your instructors

Frequently Asked Questions

If your events are recurring and at the same time, it might be easiest to use a single line item to communicate your course schedule to students

Monday & Wednesday & Friday

Feel free to type out dates as your title as a way to communicate information about specific live sessions or other events.

May 7, 2022

Schedule items can also be used to convey commitments outside of specific time slots (like weekly projects or daily office hours).

Weekly projects

Course schedule

This course builds on live workshops and hands-on projects

You’ll be interacting with other learners through breakout rooms and project teams

Course completed by

About this course

Get notified about the next cohort

Learn from live sessions