Cohort 1 has $1,000 in modal credits.  This is NOT guaranteed for Cohort 2

Cohort 1

Learn proven approaches for quickly improving AI applications.  Build AI that works better than the competition, regardless of the use-case.

Hamel Husain

ML Engineer with 20 years of experience.

Shreya Shankar

ML Systems Researcher Making AI Evaluation Work in Practice

AI Evals For Engineers & PMs

Airbnb

Google

GitHub

UC Berkeley

Stanford University

🚨 JULY 21st IS IS OUR FINAL LIVE COHORT - (We have to get back to building AI applications!) 🚨 Do you catch yourself asking any of the following questions while building AI applications? 1. How do I test applications when the outputs are stochastic and require subjective judgements? 2. If I change the prompt, how do I know I'm not breaking something else? 3. Where should I focus my engineering efforts? Do I need to test everything? 4. What if I have no data or customers, where do I start? 5. What metrics should I track? What tools should I use? Which models are best? 6. Can I automate testing and evaluation? If so, how do I trust it? If you aren't sure about the answers to these questions, this course is for you. This is a hands-on course for engineers and technical PMs. Ideal for those who are comfortable coding or vibe coding. ---WHAT TO EXPECT This course will provide you with hands-on experience. Get ready to sweat through exercises, code and data! We will meet two times a week for four weeks, with generous office hours (read below for course schedule). We will also hold office hours and host Discord community where you can communicate with us and each other. In return, you will be rewarded with skills that will set you apart from the competition by a wide margin. (see testimonials below). All sessions will be recorded and available to students asynchronously.--- COURSE CONTENT Lesson 1: Fundamentals &amp; Lifecycle LLM Application Evaluation- Why evaluation matters for LLM applications - business impact and risk mitigation- Challenges unique to evaluating LLM outputs - common failure modes and context-dependence- The lifecycle approach from development to production- Basic instrumentation and observability for tracking system behavior- Introduction to error analysis and methods for categorizing failures Lesson 2: Systematic Error Analysis- Bootstrap data through effective synthetic data generation- Annotation strategies and quantitative analysis of qualitative data- Translating error findings into actionable improvements- Avoiding common pitfalls in the analysis process- Practical exercise: Building and iterating on an error tracking system Lesson 3: Implementing Effective Evaluations- Defining metrics using code-based and LLM-judge approaches- Techniques for evaluating individual outputs and overall system performance- Organizing datasets with proper structure for inputs and reference data- Practical exercise: Building an automated evaluation pipeline Lesson 4: Collaborative Evaluation Practices- Designing efficient team-based evaluation workflows- Statistical methods for measuring inter-annotator agreement- Techniques for building consensus on evaluation criteria- Practical exercise: Collaborative alignment in breakout groups Lesson 5: Architecture-Specific Evaluation Strategies- Evaluating RAG systems for retrieval relevance and factual accuracy- Testing multi-step pipelines to identify error propagation- Assessing appropriate tool use and multi-turn conversation quality- Multi-modal evaluation for text, image, and audio interactions- Practical exercise: Creating targeted test suites for different architectures Lesson 6: Production Monitoring &amp; Continuous Evaluation- Implementing traces, spans, and session tracking for observability- Setting up automated evaluation gates in CI/CD pipelines- Methods for consistent comparison across experiments- Implementing safety and quality control guardrails- Practical exercise: Designing an effective monitoring dashboard Lesson 7: Efficient Continuous Human Review Systems- Strategic sampling approaches for maximizing review impact- Optimizing interface design for reviewer productivity- Practical exercise: Implementing a continuous feedback collection system Lesson 8: Cost Optimization- Quantifying value versus expenditure in LLM applications- Intelligent model routing based on query complexity- Practical exercise: Optimizing a real-world application for cost efficiency GUEST SPEAKERS More coming soon!---1. Teresa Torres (Product Talk) : From Noob to 5 Automated Evals in 4 Weeks (as a PM).2. Leonard Tang (Haize Labs): Scaling Inference-Time Compute for Better LLM Judges

Eliminate the guesswork of building AI applications with data-driven approaches.

Engineers &amp; Technical PMs building AI products who are interested in moving beyond proof-of-concepts.

Those interested in moving beyond vibe-checks to data driven measurements you can trust, even when outputs are stochastic or subjective.

Founders and leaders who are unsure of the failure modes of their AI applications and where to allocate resources.

Who this course is for

What you’ll get out of this course

We've tried all of them so you don't have to.

Acquire the best tools for finding, diagnosing, and prioritizing AI errors.

And how to best leverage data when you do have users.

Learn how to bootstrap with synthetic data for testing before you have users

Data flywheels ensure you have examples to draw from for prompts, tests, and fine-tuning.

Create a data flywheel for your applications that guarantees your AI will improve over time.

How can you really trust LLM-as-a-judge? How should you design them? We will show you how. We will also show you how to refine prompts, generate metadata, and other tasks with the assistance of AI.

Automate parts of your AI evaluation with approaches that allow you to actually trust and rely on them.

We will show you approaches to discover all the ways AI is not performing

Ensure your AI is aligned to your preferences, tastes and judgement.

There are an infinite number of things you can try, tests you can write, and data you can look at. We will show you a data-driven process that helps you prioritize the most important problems so you can avoid wasting time and money.

Avoid common mistakes we've seen across 35+ AI implementations.

We will provide end-to-end exercises, examples and code to make sure you come away with the skills you need. We will NOT just throw a bunch of slides at you!

Hands-On Exercises, Examples and Code

Generous office hours ensures students can ask questions about their specific issues and interests.

Personalized Instruction

Provide a brief description for each module as a way to sell the value of what students will be learning throughout the course.

First Module

Second Module

Third Module

Course syllabus

See what our students have to say

"After taking this course...I feel like a wizard who can build high quality AI agents and applications."

Can PMs also get value from this course?  Yes!  This is what PMs are saying:

Instructors are recognized experts, with hands-on experience with 30+ companies

Meet your instructors

Frequently Asked Questions

If your events are recurring and at the same time, it might be easiest to use a single line item to communicate your course schedule to students

Monday & Wednesday & Friday

Feel free to type out dates as your title as a way to communicate information about specific live sessions or other events.

May 7, 2022

Schedule items can also be used to convey commitments outside of specific time slots (like weekly projects or daily office hours).

Weekly projects

Course schedule

This course builds on live workshops and hands-on projects

You’ll be interacting with other learners through breakout rooms and project teams

Course completed by

About this course

Get notified about the next cohort

Learn from live sessions