AI Evals For Engineers & PMs

All lessons are recorded and are made exclusively available to students in the course.

Guest lectures will be made publicly available.

Time commitment is 3-4 hours per week for 4 weeks.

No. While the course is titled "AI Evals For Engineers & Product Managers," you'll learn valuable evaluation processes whether you code or not. The course covers prioritization strategies, LLM judges, evaluation approaches, and analytical techniques for identifying AI failure modes. The instructors provide generous office hours to ensure everyone c

While the course is designed to be accessible to people without extensive ML or data science backgrounds, some familiarity with AI applications and basic understanding of how LLMs work would be helpful.

The course is particularly well-suited for:

• Engineers building AI applications with limited ML experience
• Those looking to move beyond vibe checks

The course includes:

• Lifetime access to all course materials

• In-depth lessons with practical exercises

• Projects to apply what you've learned

• Access to a private community of peers

• Guided feedback and reflection

• Course certificate upon completion

This course stands out by focusing specifically on evaluation and improvement rather than just building AI systems.

Rather than just teaching tools, this course delivers the processes for systematically improving AI applications regardless of the specific technologies used.

You'll gain practical skills for:

• Building custom annotation tools for error analysis
• Creating evaluation dashboards and data viewers
• Designing targeted test suites for specific AI architectures
• Implementing automated evaluation gates in CI/CD pipelines
• Developing strategies for collecting effective human feedback

You'll learn concrete methods for evaluation with limited or no user data:

• Synthetic data generation techniques
• Creating evaluation frameworks that scale with your product
• Balancing qualitative "vibe checks" with quantitative measurements
• Low-cost approaches to collect initial feedback
• Tests that uncover issues before users experience them

The course covers evaluation for sophisticated AI architectures:

• Debugging interactions between multiple AI components
• Isolating failures in agent workflows
• Testing agent collaboration
• Measuring emergent behaviors
• Evaluating overall system reliability versus component performance

Yes. The course teaches you to:

• Translate business requirements into evaluation metrics
• Ensure AI alignment with brand values
• Measure what matters to users, not just generic metrics
• Balance competing priorities (accuracy vs. speed vs. cost)
• Communicate AI performance to stakeholders in business terms

The course is tool-agnostic but will introduce you to popular frameworks and technologies for evaluation. The focus is on teaching approaches that work regardless of your specific tech stack. You'll learn evaluation patterns that can be implemented with various tools based on your organization's needs.

Yes. The course covers evaluation strategies for different AI architectures including RAG systems, chatbots, multi-step agents, and multi-modal systems. You'll learn techniques specific to each architecture's unique challenges and failure modes.