Learn from a world-leading expert how to design and analyze trustworthy A/B tests to evaluate ideas, integrate AI/ML, and grow your business
Through multiple real examples of well-run experiments and real stories at Microsoft, Amazon, and Airbnb, you will see the humbling reality that we are terrible at assessing the values of ideas.
Trivial changes can be surprisingly useful, whereas large efforts often fail. Accelerate innovation by building Minimum Viable Products and Features (MVPs) and make the organization evidence-based and humbler, as it adopts and learns to use evidence from the gold standard in science: the controlled experiment.
You will understand the challenges in designing and running trustworthy controlled experiments, or A/B tests, including the importance of the Overall Evaluation Criterion (OEC), scaling, pitfalls, and Twyman's law.
The 2nd week covers additional topics, some more technical, including cultural challenges, institutional memory, maturity model, observational causal studies, offline evaluations, AI/Machine learning and triggering, Bayesian vs. Frequentist, Scaling, Build vs. But, Challenges, and requested topics.
If you're interested in a less-technical introduction to A/B testing that covers mostly the first week of this class, see https://bit.ly/ABEssentialsCourseRKFS
Data science managers and scientists will be able to design and interpret the experiment results in a trustworthy manner
Program managers focused on growth, revenue, conversions, and prioritization will understand how to provide the org with robust clear metric
Engineering leaders will be able to make the organizations more data-driven and efficient with fewer severe incidents through A/B tests
You will hear multiple real memorable stories and examples, many that don't make it to books or articles. These were chosen from over 20 years of experimentation. You'll have the data to show that the poor success rate is documented across multiple organizations; expected it!
Understand key concepts like causality, hierarchy of evidence, and key organizational tenets required for effective experimentation.
Designing metrics is hard. There is a hierarchy of metrics and perverse incentives. The most important metrics comprise of the OEC - The Overall Evaluation Criterion. We will look at good and bad examples.
Getting numbers is easy; getting numbers you can trust is hard. You'll understand common pitfalls and how to design reliable and trustworthy tests.
Learn about the cultural challenges, the humbling results (most ideas fail, pivoting, iterating, learning), institutional memory, ideation, prioritization, experimentation platforms
When building AI or machine learning models, using A/B testing and triggering to evaluate the models that were built offline based on historical data
When you can't run an A/B test, quasi-experimentation methods and the risks of observational causal studies
What are key challenges and open questions in the field
If there is something specific you want to cover, there is time allocated for topics voted by the audience to discuss
The course focuses on developing the intuition and common misunderstandings, without the details of the statistics, which you can find in many books. We cover p-values, statistical power, and triggering.
You can go as technical as you want in the Q&A and community discussions
Accelerating Innovation with AB Testing
Feb 5—Feb 11
Mon, Feb 5, 4:00 PM - 6:00 PM UTC
Tue, Feb 6, 4:00 PM - 6:00 PM UTC
Thu, Feb 8, 4:00 PM - 6:00 PM UTC
Feb 12—Feb 15
Mon, Feb 12, 4:00 PM - 6:00 PM UTC
Thu, Feb 15, 4:00 PM - 6:00 PM UTC
Thu, Feb 29, 4:00 PM - 5:00 PM UTC
Open to all alumni from all cohorts
Manuel de Francisco Vera
Ronny Kohavi was an executive at Amazon, Microsoft, and Airbnb and has over 20 years of experience running A/B tests and leading experimentation teams. He loves to teach, and his papers have over 55,000 citations. He co-authored the best-selling book: Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing (with Diane Tang and Ya Xu), which is a top-10 data mining book on Amazon. He is the most viewed writer on Quora's A/B testing and received the Individual Lifetime Achievement Award for Experimentation Culture in Sept 2020.
Ronny holds a PhD in Machine Learning from Stanford University.
See more at http://www.kohavi.com
8-10AM Pacific Time
Three x 2-hour sessions in week 1 on Monday, Tuesday, and Thursday
8-10AM Pacific Time
Two x 2-hour sessions in week 2 on Monday and Thursday
15 minutes after each session
We will review multiple real A/B tests
Deep dive design and analysis of two A/B tests
We will deep dive into the full lifecycle of designing an A/B test to answer a hypothesis and analyze the results
Learn with a cohort of peers
Join a community of like-minded people who want to learn and grow alongside you
Missing anything? We have allocated time to suggest topics, collect votes, and discuss them on the last session
Sign up to be the first to know about course updates.