Free Lesson
AI Evals for Claude Code Analytics Output
Part of The AI Evaluation Handbook
60 min
Jun 17, 2026 11:00 AM
Virtual (Zoom)
In this video
What you'll learn
See how an agentic analysis actually breaks
The path from question to SQL to recommendation, and where confident-but-wrong answers slip in
Learn the variety of ways to check any AI analysis
and how to match the effort to the stakes
Watch an AI Eval reliability test live
Run a question several times and measure what's stable vs what drifts (no answer key required)
Walk away with a one-page checklist you can use on Monday
Something you can immediately apply to you day-to-day
Why this topic matters
AI will now run your whole analysis in seconds, confident, polished, and sometimes completely wrong. Getting the answer stopped being the hard part; knowing whether to trust it is the new skill. This session teaches the high-level framework for validating agentic analytics from scratch: what it is, where it breaks, and the few ways to check any output.
You'll learn from

Shane Butler
Principal Data Scientist at Ontra

Sravya Madipalli
Senior DS Leader (Ex-Microsoft)

Hai Guan
Head of Data at Ontra, Ex-LinkedIn
Previously at Stripe, Nextdoor, PwC