AI Evals vs. Product Metrics: A 30min Crash Course

Hosted by Stella Liu

Fri, Jan 9, 2026

4:00 PM UTC (30 minutes)

Virtual (Zoom)

Free to join

50 students

Invite your network

Go deeper with a course

AI Evals and Analytics Playbook
Stella Liu and Amy Chen
View syllabus

What you'll learn

When and why to align AI metrics with product goals

Learn how to translate between business outcomes and concrete AI metrics

Evals vs. product metrics across product lifecycle

How to use evals pre-release and analytics post-release to guide shipping, monitoring, and iteration

Evals and product metrics for ChatGPT vs. for CX chatbot

Real-world comparison of metrics for products like ChatGPT vs. your own customer service chatbot

Build an Evals + Analytics feedback loop

How to combine product KPIs and evals dashboards into a simple, repeatable improvement system using a playbook

Why this topic matters

AI teams often chase generic “hallucination” scores or vendor dashboards that look impressive but fail to move core product and business metrics. Understanding how evals and product analytics work together helps you focus on what truly matters: delivering reliable, valuable AI product experiences that users trust and pay for.

You'll learn from

Stella Liu

Head of Applied Scientist, AI Evals

Stella Liu is an AI Evaluation practitioner and researcher, specializing in frameworks for large language models and AI-powered products.


Since 2023, she has led real-world AI evaluation projects in EdTech, where she established the first AI product evaluation framework for Higher Education and continues to advance research on the safe and responsible use of AI. Her work combines academic rigor with hands-on product experience, bringing proven evaluation methods into both enterprise and educational contexts.


Earlier in her career, Stella worked at Shopify and Carvana, where she built large-scale data-driven automation systems that powered product innovation and operational efficiency at scale.


Stella also writes an AI Evals newsletter on Substack: https://datasciencexai.substack.com/


Follow Stella on LinkedIn

Previously at

Shopify.com
Carvana
Harvard University
UT Austin

Sign up to join this lesson

By continuing, you agree to Maven's Terms and Privacy Policy.