AI Evals and Analytics

Cohort 3

Cohort 1

Cohort 2

Cohort 2 - skipped

Cohort 4

Cohort 5

Cohort 6

Cohort 7

Product managers who drive strategy, development, and launch of products.

Engineers and engineering leaders who build, deploy, and scale software systems.

Data scientists who apply statistical and ML techniques to extract insights from data.

Measuring and evaluating AI model performance, quality, and reliability for production use cases.

Applying AI tools and techniques to professional workflows — assign alongside more specific AI topics.

Using data and metrics to measure performance, run experiments, and inform decisions.

Learn how to evaluate, ship, and monitor GenAI products that perform reliably in production.

Head of AI Applied Science

Stella Liu

Cofounder, AI Evals & Analytics

Amy Chen

Cofounder at AI Evals & Analytics

AI Evals and Analytics Playbook

Product Manager

Accessibility Technical Manager

Technology Advisor

Data Scientist

New Grad

Founder

AI Engineer/ AI Scientist

Staff Machine Learning Engineer

AI product manager

Shopify

Carvana

Harvard University

System1

Uptake

Seasalt.ai

UCLA

University of Washington

<blockquote>Building AI is easier than ever. Building AI that is reliable, safe, and trusted is not.AI evaluations help teams measure quality, detect failures, and build guardrails around your GenAI systems that will survive in the real world.</blockquote><blockquote>🔹</blockquote><h3>Who this course is for</h3>Data scientists, PMs and QA/QE professionals who:<ul><li>have no prior background in AI evals, or</li><li>have tried LLM-as-a-judge but want a clearer framework.</li></ul><blockquote>🔹How the course works</blockquote><ul><li>Self-paced lectures you can watch anytime</li><li>Hands-on exercises using a pre-built GenAI product</li><li>Dedicated office hours for questions and guidance</li></ul><blockquote>🔹</blockquote><h3>What makes this course different</h3><ul><li>Playbook-driven approach</li><li>Evaluation-driven development</li><li>Product-focused</li><li>Hands-on experience</li><li>Flexible learning + expert support</li></ul>🔹Curious about the ideas behind this course?You can explore related topics in the <a target="_blank" rel="noopener noreferrer nofollow" href="https://datasciencexai.substack.com/">Data Science × AI Substack</a>, where we share perspectives on evaluating real GenAI products.🔹💰 Purchasing Power Parity pricing adjustments upon request. ✉️ Email us: StellaxAmy@gmail.com

Learn How to Evaluate GenAI Products Confidently in 3 Weeks

Execution-ready Testing & Monitoring Playbook

GenAI Risk Management Strategy

Hands-On AI Evals Exercise

Product LeadersBuilding GenAI products and need a clear playbook for evaluation frameworks, success metrics, and shipping with confidence.

Data ScientistsLeveling up with AI product skills: learn AI evals, LLM-as-a-judge, and design metrics for GenAI systems.

QA and Test EngineerAdapting traditional testing practices to GenAI systems.

The Analytics Mindset Behind Effective AI Evals

Rogério Chaves

Collaborative AI Evals with Human Feedback

Ravin Kumar

Evals for Voice AI: Learnings from Google Evals Team

Eddie Landesberg

Calibrate LLM-as-a-judge for Real-world Impact

AI Evals vs. Product Metrics: A 30min Crash Course

Pragmatic AI Evals: Beyond Vendor Promises

A Preview of Our AI Evals Playbook and Framework

AI Evals vs. Product Metrics

Bring AI Evals to Product Leadership

Who this session is for

What we can cover

What you will leave with

One-hour expert guidance on AI eval strategy, team setup, stakeholder alignment, and implementation.

AI Evals Researcher & Practitioner | Head of AI Applied Science

AI Evals Strategy & Implementation Consulting

Learn directly from AI Evals and Analytics

AI Evals and Analytics

Courses

AI Evals and Analytics Playbook

Lightning Lessons

The Analytics Mindset Behind Effective AI Evals

Collaborative AI Evals with Human Feedback

Evals for Voice AI: Learnings from Google Evals Team

Calibrate LLM-as-a-judge for Real-world Impact

AI Evals vs. Product Metrics: A 30min Crash Course

Pragmatic AI Evals: Beyond Vendor Promises

Services

AI Evals Strategy & Implementation Consulting

Resources

A Preview of Our AI Evals Playbook and Framework

AI Evals vs. Product Metrics

Bring AI Evals to Product Leadership

Calibrate LLM-as-a-judge for Real-world Impact