Do you catch yourself asking any of the following questions while building AI applications?

1. How do I test applications when the outputs are stochastic and require subjective judgements?

2. If I change the prompt, how do I know I'm not breaking something else?

3. Where should I focus my engineering efforts? Do I need to test everything?

4. What if I have no data or customers, where do I start?

5. What metrics should I track? What tools should I use? Which models are best?

6. Can I automate testing and evaluation? If so, how do I trust it?

If you aren't sure about the answers to these questions, this course is for you.

This is a hands-on course for engineers and technical PMs. Ideal for those who are comfortable coding or vibe coding.

---

WHAT TO EXPECT

This course will provide you with hands-on experience. Get ready to sweat through exercises, code and data! We will meet two times a week for four weeks, with generous office hours (read below for course schedule).

We will also hold office hours and host Discord community where you can communicate with us and each other. In return, you will be rewarded with skills that will set you apart from the competition by a wide margin. (see testimonials below). All sessions will be recorded and available to students asynchronously.

---

COURSE CONTENT

Lesson 1: Fundamentals & Lifecycle LLM Application Evaluation

- Why evaluation matters for LLM applications - business impact and risk mitigation

- Challenges unique to evaluating LLM outputs - common failure modes and context-dependence

- The lifecycle approach from development to production

- Basic instrumentation and observability for tracking system behavior

- Introduction to error analysis and methods for categorizing failures

Lesson 2: Systematic Error Analysis

- Bootstrap data through effective synthetic data generation

- Annotation strategies and quantitative analysis of qualitative data

- Translating error findings into actionable improvements

- Avoiding common pitfalls in the analysis process

- Practical exercise: Building and iterating on an error tracking system

Lesson 3: Implementing Effective Evaluations

- Defining metrics using code-based and LLM-judge approaches

- Techniques for evaluating individual outputs and overall system performance

- Organizing datasets with proper structure for inputs and reference data

- Practical exercise: Building an automated evaluation pipeline

Lesson 4: Collaborative Evaluation Practices

- Designing efficient team-based evaluation workflows

- Statistical methods for measuring inter-annotator agreement

- Techniques for building consensus on evaluation criteria

- Practical exercise: Collaborative alignment in breakout groups

Lesson 5: Architecture-Specific Evaluation Strategies

- Evaluating RAG systems for retrieval relevance and factual accuracy

- Testing multi-step pipelines to identify error propagation

- Assessing appropriate tool use and multi-turn conversation quality

- Multi-modal evaluation for text, image, and audio interactions

- Practical exercise: Creating targeted test suites for different architectures

Lesson 6: Production Monitoring & Continuous Evaluation

- Implementing traces, spans, and session tracking for observability

- Setting up automated evaluation gates in CI/CD pipelines

- Methods for consistent comparison across experiments

- Implementing safety and quality control guardrails

- Practical exercise: Designing an effective monitoring dashboard

Lesson 7: Efficient Continuous Human Review Systems

- Strategic sampling approaches for maximizing review impact

- Optimizing interface design for reviewer productivity

- Practical exercise: Implementing a continuous feedback collection system

Lesson 8: Cost Optimization

- Quantifying value versus expenditure in LLM applications

- Intelligent model routing based on query complexity

- Practical exercise: Optimizing a real-world application for cost efficiency

---

FREE CREDITS

Each student will get $1,000 of of free Modal (https://modal.com/) compute credits.

---

GUEST SPEAKERS

1. Kwindla Kramer: Founder of pipecat, an OSS framework for voice and conversational AI. - How to eval voice agents.

2. JJ Allaire: Creator of OSS tool Inspect, an eval toolset used by Anthropic, Groq and other leading companies.

3. Eugene Yan: Principal Applied Scientist at Amazon. "An LLM‑as‑Judge Won't Save Your Product—Fixing Your Process Will"

4. Aarush Sah: Head of Evals at Groq

5. Benjamin Clavié: Creator of OSS project RAGatouille and rerankers: RAG mistakes & how to diagnose them.

6. Isaac Flath: Creator of MonsterUI and FastHTML core contributor - Building custom annotation tools for error analysis with FastHTML

7. Mikyo King: Creator of OSS Evals framework Phoneix.

8. Bryan Bischof: Head of AI @ Theory & formerly Hex, W&B and StitchFix. Failure funnels - an analytical framework for simplifying Agent evaluations.

9. Brooke Hopkins: Founder of Coval, simulation & evals for voice and chat agents

10. Alex Volkov: Devolper advocate @ Weights & Biases- "Reasoning models and LLM as a judge"

11. Aman Khan: AI PM @ Arize