A/B Testing Replication Crisis: Lessons from 10 A/B tests

Hosted by Ronny Kohavi

Wed, Mar 4, 2026

6:00 PM UTC (45 minutes)

Virtual (Zoom)

Free to join

132 students

Invite your network

Go deeper with a course

Accelerating Innovation with A/B Testing
Dr. Ronny Kohavi
View syllabus

What you'll learn

How to design a trustworthy A/B tests

How we designed the 10 A/B tests to evaluate four patterns: choosing the metrics and duration for proper power.

Most claims of large improvements are highly exaggerated

While we expected to see exaggerations, our results showed surprisingly small effects.

Lessons

Key lessons from the Trustworthy A/B Patterns community project.

Why this topic matters

Claims of large lifts in A/B tests are widespread, yet many are only supported by small online experiments. The Trustworthy A/B Patterns community project (https://bit.ly/trustworthyABPatterns) was started in June 2024 by Kohavi, Linowski, and Vermeer to provide realistic assessment of several patterns with large-scale A/B tests.

You'll learn from

Ronny Kohavi

Technical fellow and VP at Microsoft & Airbnb, co-author of A/B testing book

Ronny Kohavi was an executive at Amazon, Microsoft, and Airbnb and has over 20 years of experience running A/B tests and leading experimentation teams. He loves to teach, and his papers have over 65,000 citations. He co-authored the best-selling book: Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing (with Diane Tang and Ya Xu), which is a top-10 data mining book on Amazon. He is the most viewed writer on Quora's A/B testing and received the Individual Lifetime Achievement Award for Experimentation Culture in Sept 2020.

Ronny holds a PhD in Machine Learning from Stanford University.

Previously at Microsoft, Airbnb, Amazon

Microsoft
Airbnb
Amazon

Sign up to join this lesson

By continuing, you agree to Maven's Terms and Privacy Policy.