Trust Your AI Analytics: Know When the Number Is Right

Part of Agentic Analytics: Validation and Context

Hosted by Shane Butler, Sravya Madipalli, and Hai Guan

Wed, Oct 14, 2026

5:00 PM UTC (1 hour)

Virtual (Zoom)

Free to join

Invite your network

Go deeper with a course

Automate AI Evals with Claude Code
Shane Butler
View syllabus

What you'll learn

Spot the confident but wrong answer

Learn the tells of a plausible number that does not hold up, so a fluent AI reply does not fool you.

Run the checks that need no answer key

Use order-of-magnitude, spot-check, and cross-total tests that catch errors even when you cannot verify the exact figure

Match the rigor to the stakes

Scale your checking to the decision: a quick gut-check for a Slack reply, a full audit before a board number.

Decide when the number is good enough to ship

Set a clear bar for "trust it and move" versus "dig deeper," so you stop second-guessing every result.

Why this topic matters

AI output looks the same whether it is right or wrong. It is fluent and confident either way, and that is the trap. Getting an answer stopped being the hard part. The hard part is knowing whether to trust it before it lands in a deck or a decision. This session is the quick checks: spot the confident wrong result, run tests that need no answer key, and know when a number is good enough to act on.

You'll learn from

Shane Butler

Co-founder, AI Analyst Lab

Shane Butler is a Co-Founder of the AI Analyst Lab. Previously he led evaluation strategy for AI product development in the legal tech domain. He has more than ten years of experience in product data science and causal inference, with prior roles at Stripe, Nextdoor, and PwC. Shane is also the co-host of the AI podcast Data Neighbor, where he interviews product, data, and engineering leaders who are pioneering the next generation of data science and analytics in an AI-driven landscape.

Sravya Madipalli

Senior DS Leader (Ex-Microsoft)

Sravya Madipalli is a Senior Manager of Data Science with 14+ years of experience helping teams make better decisions with data. She has built and led data science and product analytics teams at Microsoft, eBay, Nextdoor, and Superhuman (prev. Grammarly), working closely with product, engineering, marketing, and leadership. Her expertise spans experimentation, metrics design, modeling, analytics, and translating complex user behavior into clear, actionable insights.

Hai Guan

Head of Data at Ontra, Ex-LinkedIn

Hai Guan leads the data organization at Ontra, the leading legal tech AI solutions for private markets. He previously led Data Science & Analytics at LinkedIn, Nextdoor, Pinterest, and Meta. He's spent a decade teaching product development teams how to ask questions that actually drive decisions—and now teaches how to combine that judgment with AI to move 10x faster.

See all products from AI

Sign up to join this lesson

By continuing, you agree to Maven's Terms and Privacy Policy.