Shane Butler
Sravya Madipalli
Hai Guan
Free Lesson

Experimentation 101: Run an A/B Test with AI

Part of Build Your AI Product Analyst

60 min
Oct 21, 2026 1:00 PM

What you'll learn

Frame a testable hypothesis

Turn a vague "let's try this" into a clear statement of what changes, for whom, and what metric should move.

Size the test so the result means something

Work out how many users and how long you need before you start, so you are not reading noise.

Read the result without fooling yourself

Spot false positives, peeking, and stat-sig traps, and know when a lift is real enough to trust.

Decide ship, hold, or roll back

Translate the numbers into a clear call, with the tradeoffs and risks spelled out.

You'll learn from

Shane Butler

Shane Butler

Co-founder, AI Analyst Lab

Sravya Madipalli

Sravya Madipalli

Senior DS Leader (Ex-Microsoft)

Hai Guan

Hai Guan

Head of Data at Ontra, Ex-LinkedIn

See all products from AI
Get free access