Quick Introduction to A/B Testing

Hosted by Ronny Kohavi

Share this lesson

686 views

What you'll learn

The pros and cons of A/B testing

When should you run A/B tests and why?

Necessary ingredients

How many users do you need (power)? What are you optimizing for (the OEC)?

Humbling statistics

Most experiments fail to improve the metrics they were designed to improve. Plan to fail fast and pivot often.

Twyman's Law

That A/B test that improved purchases by 20% (e.g., by rounding the buttons) is bogus. Be skeptical.

Questions? Add and vote at

https://bit.ly/QuickIntoABTQuestions

Why this topic matters

A/B testing has transformed many organizations and helped accelerate innovation.

You'll learn from

Ronny Kohavi

Technical fellow and VP at Microsoft & Airbnb, co-author of A/B testing book

Ronny Kohavi was an executive at Microsoft, Airbnb, and Amazon, and has over 20 years of experience running A/B tests and leading experimentation teams. He loves to teach, and his papers have over 60,000 citations. He co-authored the best-selling book: Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing (with Diane Tang and Ya Xu), which is a top-10 data mining book on Amazon. He is a most viewed writer on Quora's A/B testing and received the Individual Lifetime Achievement Award for Experimentation Culture in Sept 2020.


Ronny holds a PhD in Machine Learning from Stanford University.

See more at http://www.kohavi.com


Questions? Add and vote at https://bit.ly/QuickIntoABTQuestions

Microsoft, Amazon, and Airbnb

Microsoft
Airbnb
Amazon
© 2024 Maven Learning, Inc.