Free Lesson

Evaluate AI Agents with Simulated Users

Hosted by Stella Liu

Fri, Aug 7, 2026

7:00 PM UTC (30 minutes)

Virtual (Zoom)

Free to join

Invite your network

Go deeper with a course

AI Evals and Analytics Playbook
Stella Liu and Amy Chen
View syllabus

What you'll learn

Understand why user simulation matters

Why do we need to simulate agent behavior across realistic, multi-turn conversations and tool calls.

Turn workflows into test scenarios

Define user context, data states, policies, and measurable success criteria.

Build realistic user simulations

An introduction to available tools for simulating dynamic, tool-aware user interactions.

Why this topic matters

Most production agents do not operate on a single prompt. They work through evolving conversations, make tool calls, request information, encounter incomplete or conflicting context, and must follow policies along the way. In this lesson, we’ll demonstrate how user simulation provides a practical way to test agent behavior using tau2-bench.

You'll learn from

Stella Liu

Head of Applied Scientist, AI Evals

Stella Liu is an AI Evaluation practitioner and researcher, specializing in frameworks for large language models and AI-powered products.


Since 2023, she has led real-world AI evaluation projects in EdTech, where she established the first AI product evaluation framework for Higher Education and continues to advance research on the safe and responsible use of AI. Her work combines academic rigor with hands-on product experience, bringing proven evaluation methods into both enterprise and educational contexts.


Earlier in her career, Stella worked at Shopify and Carvana, where she built large-scale data-driven automation systems that powered product innovation and operational efficiency at scale.


Stella also writes an AI Evals newsletter on Substack: https://datasciencexai.substack.com/


Follow Stella on LinkedIn

Previously at

Shopify.com
Carvana
Harvard University
UT Austin
See all products from AI Evals and Analytics

Sign up to join this lesson

By continuing, you agree to Maven's Terms and Privacy Policy.