Pressure-test any AI analysis

Part of The AI Evaluation Handbook

Hosted by Shane Butler, Sravya Madipalli, and Hai Guan

829 students

In this video

What you'll learn

Find or forge a reference to check an AI analysis against

The spectrum, and which fits your company

Handle the novel question that has no answer key

With checks you can run on any output

Know what each check actually catches

And where it falsely passes

Join AI Builders Slack Community. Link: bit.ly/ai-connect

Learn Agentic Analytics workflows and AI Eval design. Get feedback on what you’re building from other AI builders.

Decide when to act, when to dig,

And when to abstain

Why this topic matters

AI analytics tools are confident even when they're wrong, and "is this right?" is the question they all raise. Knowing means checking against ground truth. Sometimes you have a reference (there's a spectrum of ways to get one, and which fits depends on your company), and sometimes you genuinely don't (the novel question, which is exactly where AI is most useful). This lesson covers both.

You'll learn from

Shane Butler

Co-Founder AI Analsyt Lab (Ex-Stripe)

Shane Butler is a Co-Founder of AI Analyst Lab. He previously led evaluation strategy for AI product development in the legal tech domain. He has more than ten years of experience in product data science and causal inference, with prior roles at Stripe, Nextdoor, and PwC. His current work focuses on practical, end-to-end methods for evaluating AI features in production. Shane is also the co-host of the AI podcast Data Neighbor, where he interviews product, data, and engineering leaders who are pioneering the next generation of data science and analytics in an AI-driven landscape.

Sravya Madipalli

Senior DS Leader (Ex-Microsoft)

Sravya Madipalli is a Senior Manager of Data Science with 14+ years of experience helping teams make better decisions with data. She has built and led data science and product analytics teams at Microsoft, eBay, Nextdoor, and Superhuman (prev. Grammarly), working closely with product, engineering, marketing, and leadership. Her expertise spans experimentation, metrics design, modeling, analytics, and translating complex user behavior into clear, actionable insights.

Hai Guan

Head of Data at Ontra, Ex-LinkedIn

Hai Guan leads the data organization at Ontra, the leading legal tech AI solutions for private markets. He previously led Data Science & Analytics at LinkedIn, Nextdoor, Pinterest, and Meta. He's spent a decade teaching product development teams how to ask questions that actually drive decisions—and now teaches how to combine that judgment with AI to move 10x faster.

Stripe
Nextdoor
eBay
Microsoft
LinkedIn
See all products from AI

Go deeper with a course

Agentic Analytics 201: Validation & Context Management
Shane Butler, Sravya Madipalli, and Hai Guan
View syllabus