AI Evals For Engineers & PMs

Understand instrumentation and observability for tracking system behavior.

Learn approaches for generating synthetic data to maximize error discovery and bootstrap product development.

Understand how to choose the right tools and vendors for you, with deep dives into the most popular solutions in the evals space.

Apply data analysis techniques to rapidly find systematic issues in your product regardless of the use case.

Master the processes and tools to annotate and analyze data quickly and efficiently.

Learn how to analyze agentic systems (tool calls, RAG, etc.) to quickly identify systematic patterns and errors.

Create evals that are customized to your product and provide immediate value, NOT generic off the shelf evals (which do not work).

Align evals with stakeholders & domain experts that allow you to scientifically trust the evals.

Create high-quality LLM-as-a-judge and code based evals with a systematic, iterative process.

Learn how to measure & debug RAG systems for retrieval relevance and factual accuracy.

Understand how to tame multi-step pipelines to identify error propagation and root-causes of errors quickly.

Master techniques that apply to multi-modal settings, including text, image, and audio interactions.

Learn how to set up automated evaluation gates in CI/CD pipelines.

Understand methods for consistent comparison across experiments, including how to prepare and maintain datasets to prevent overfitting.

Implement safety and quality control guardrails.

Develop a strong intuition of when to write an eval, and when NOT write an eval.

Learn how to design interfaces to remove friction from reviewing data and collect higher quality data with less effort.

Learn how to avoid common pitfalls surrounding team organization, collaboration, responsibilities, tools, automation, and metrics.