2 Weeks
·Cohort-based Course
Learn from videos and live sessions with a world-leading expert how to design and analyze trustworthy A/B tests to accelerate innovation
Hosted by
Dr. Ronny Kohavi
Technical fellow and VP at Microsoft & Airbnb, co-author of A/B testing book
Course overview
Through multiple real examples and real stories, you will see the humbling reality of surprising experiments that ran at companies including Microsoft, Amazon, and Airbnb.
You will understand the challenges in designing and running trustworthy controlled experiments, or A/B tests, including the importance of the Overall Evaluation Criterion (OEC), scaling, pitfalls, and Twyman's law.
This is a shorter and less technical version of the flagship course https://bit.ly/ABClassRKFlagship. You will view videos at your own pace and attend two interactive one-hour sessions.
01
Anyone interested in A/B testing that is too busy to take the flagship course https://bit.ly/ABClassRKFlagship, or prefers async learning
02
Engineering leaders will be able to make the organizations more data-driven and efficient with fewer severe incidents
03
Program managers focused on growth, revenue, conversions, and prioritization will understand how to design robust clear metrics
04
Data science managers and scientists who do not need or want the technical details, but rather looking for the high-level overview
You will hear multiple real memorable stories and examples, many that don't make it to books or articles. These were chosen from over 20 years of experimentation: motivation, pushback, success/failure rates
Understand the key benefits of controlled experiments, or A/B tests, including causality, hierarchy of evidence, surprising examples, metrics, interpreting results, trust and pitfalls, Twyman's law, and A/A tests
Designing metrics is hard. There is a hierarchy of metrics and perverse incentives. The most important metrics comprise of the OEC - The Overall Evaluation Criterion. We will look at good and bad examples.
Getting numbers is easy; getting numbers you can trust is hard. We'll discuss pitfalls and how to build a reliable and trustworthy experimentation system.
Innovation & AB Testing - the Essentials
Dylan Lewis
Ryan Lucht
Pavan Gangisetty
Han Dong
Ishan Goel
Deborah O'Malley
Jialin Huang
Scott Rome
Aaron Gasperi
Manuel de Francisco Vera
Ronny Kohavi was an executive at Amazon, Microsoft, and Airbnb and has over 20 years of experience running A/B tests and leading experimentation teams. He loves to teach, and his papers have over 60,000 citations. He co-authored the best-selling book: Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing (with Diane Tang and Ya Xu), which is a top-10 data mining book on Amazon. He is the most viewed writer on Quora's A/B testing and received the Individual Lifetime Achievement Award for Experimentation Culture in Sept 2020.
Ronny holds a PhD in Machine Learning from Stanford University.
See more at http://www.kohavi.com
01
Videos: Introduction and Motivation
Videos (view on your own asynchronously):
02
Videos: A/B Testing: tenets, advantage & issues, example
Videos (view on your own asynchronously):
03
Videos: Metrics Taxonomy & the OEC
04
Monday: Live session
05
Videos: Designing and Interpreting Experiments
06
Videos: Twyman's Law, Trust, and idea generation and prioritization
07
Videos: Additional Topics
08
Thursday: Live session
Watch videos on your own
Attend live session at 8AM or 5PM PST
Interested in reading chapter 1 of my book: Trustworthy Online Controlled Experiments : A Practical Guide to A/B Testing?
Nov 2023
$999 USD
Dates
Nov 6—17, 2023
Payment Deadline
Nov 6, 2023
Real-world examples
We will review multiple real A/B tests
Deep dive design and analysis of an A/B test - end to end
We will deep dive into the full lifecycle of designing an A/B test to answer a hypothesis and analyze the results
Learn with a cohort of peers
Join a community of like-minded people who want to learn and grow alongside you
Requested topics
Missing anything? Suggest topics, vote, and we will discuss in the last live session
Nov 2023
$999 USD
Dates
Nov 6—17, 2023
Payment Deadline
Nov 6, 2023