Agentic Experiment Design & Analysis

Hosted by Sravya Madipalli, Hai Guan, and Shane Butler

Fri, Jun 5, 2026

7:00 PM UTC (1 hour)

Virtual (Zoom)

Free to join

Invite your network

Go deeper with a course

Automate AI Evals with Claude Code
Shane Butler
View syllabus

What you'll learn

Frame experiments using the Agentic Loop

Apply the 4 stages (Frame, Power, Validate, Decide) to design tests that hold up to scrutiny.

Catch the validity gaps that ship bad decisions

Spot sample ratio mismatch, novelty effects, and underpowered tests before the readout meeting.

Use AI agents to design and analyze A/B tests

Apply a design helper prompt and watch Claude Code produce a full experiment readout from raw data.

Why this topic matters

Most experiment readouts ship bad decisions: p-values without validity checks, no segment analysis, vague recommendations. AI makes it easier to crunch numbers, which makes it easier to crunch the wrong ones. Learn the Agentic Experiment Loop to design, validate, and analyze tests, then see how AI agents accelerate every stage without replacing judgment.

You'll learn from

Sravya Madipalli

Senior DS Leader (Ex-Microsoft)

Sravya Madipalli is a Senior Manager of Data Science with 14+ years of experience helping teams make better decisions with data. She has built and led data science and product analytics teams at Microsoft, eBay, Nextdoor, and Superhuman (prev. Grammarly), working closely with product, engineering, marketing, and leadership. Her expertise spans experimentation, metrics design, modeling, analytics, and translating complex user behavior into clear, actionable insights.

Hai Guan

Head of Data at Ontra, Ex-LinkedIn

Hai Guan leads the data organization at Ontra, the leading legal tech AI solutions for private markets. He previously led Data Science & Analytics at LinkedIn, Nextdoor, Pinterest, and Meta. He's spent a decade teaching product development teams how to ask questions that actually drive decisions—and now teaches how to combine that judgment with AI to move 10x faster.

Shane Butler

Principal Data Scientist at Ontra

Shane Butler is a Principal Data Scientist at Ontra, where he leads evaluation strategy for AI product development in the legal tech domain. He has more than ten years of experience in product data science and causal inference, with prior roles at Stripe, Nextdoor, and PwC. Shane is also the co-host of the AI podcast Data Neighbor, where he interviews product, data, and engineering leaders who are pioneering the next generation of data science and analytics in an AI-driven landscape.

See all products from AI Analyst Lab

Sign up to join this lesson

By continuing, you agree to Maven's Terms and Privacy Policy.