4.7
(66 ratings)
2 Weeks
·Cohort-based Course
Learn from a world-leading expert how to design and analyze trustworthy A/B tests to evaluate ideas, integrate AI/ML, and grow your business
4.7
(66 ratings)
2 Weeks
·Cohort-based Course
Learn from a world-leading expert how to design and analyze trustworthy A/B tests to evaluate ideas, integrate AI/ML, and grow your business
Course overview
Through multiple real examples of well-run experiments and real stories at Microsoft, Amazon, and Airbnb, you will see the humbling reality that we are terrible at assessing the values of ideas.
Trivial changes can be surprisingly useful, whereas large efforts often fail. Accelerate innovation by building Minimum Viable Products and Features (MVPs) and make the organization evidence-based and humbler, as it adopts and learns to use evidence from the gold standard in science: the controlled experiment.
You will understand the challenges in designing and running trustworthy controlled experiments, or A/B tests, including the importance of the Overall Evaluation Criterion (OEC), scaling, pitfalls, and Twyman's law.
The 2nd week covers additional topics, some more technical, including cultural challenges, institutional memory, maturity model, observational causal studies, offline evaluations, AI/Machine learning and triggering, Bayesian vs. Frequentist, scaling, build vs. buy, challenges, and requested topics.
01
Data science managers and scientists will be able to design and interpret the experiment results in a trustworthy manner
02
Program managers focused on growth, revenue, conversions, and prioritization will understand how to provide the org with robust clear metric
03
Engineering leaders will be able to make the organizations more data-driven and efficient with fewer severe incidents through A/B tests
Understand and internalize the humbling reality that we are poor at assessing the values of ideas: most ideas fail!
You will hear multiple real memorable stories and examples, many that don't make it to books or articles. These were chosen from over 20 years of experimentation. You'll have the data to show that the poor success rate is documented across multiple organizations; expected it!
Understand the key advantages and limitations of A/B testing
Understand key concepts like causality, hierarchy of evidence, and key organizational tenets required for effective experimentation.
Learn how to design metrics and the Overall Evaluation Criterion
Designing metrics is hard. There is a hierarchy of metrics and perverse incentives. The most important metrics comprise of the OEC - The Overall Evaluation Criterion. We will look at good and bad examples.
Learn how to designing trustworthy A/B tests
Getting numbers is easy; getting numbers you can trust is hard. You'll understand common pitfalls and how to design reliable and trustworthy tests.
Learn about the cultural challenges
Learn about the cultural challenges, the humbling results (most ideas fail, pivoting, iterating, learning), institutional memory, ideation, prioritization, experimentation platforms
Learn about the relationship to AI and Machine Learning Modeling
When building AI or machine learning models, using A/B testing and triggering to evaluate the models that were built offline based on historical data
Learn about complementary quasi-experimental techniques
When you can't run an A/B test, quasi-experimentation methods and the risks of observational causal studies
Learn about existing challenges
What are key challenges and open questions in the field
Learn about YOUR topic of interest
If there is something specific you want to cover, there is time allocated for topics voted by the audience to discuss
Technical to your needs
The course focuses on developing the intuition and common misunderstandings, without the details of the statistics, which you can find in many books. We cover p-values, statistical power, and triggering.
You can go as technical as you want in the Q&A and community discussions
Accelerating Innovation with AB Testing
Dec
2
Dec
3
Dec
5
Jan
9
4.7
(66 ratings)
Not satisfied after attending the first session and before session 2 starts? Get a full refund
Dylan Lewis
Ryan Lucht
Pavan Gangisetty
Han Dong
Scott Theisen
Sharath Bulusu
James Niehaus
Ishan Goel
Jakub Linowski
Deborah O'Malley
Aaro Wroblewski
Jialin Huang
Scott Rome
Aaron Gasperi
Jessica Porges
Haiyan Chen
Emma Ding
Manuel de Francisco Vera
Markus Wiggering
Sorin Tarna
Gabriel Rodriguez
Top: Companies sorted by approximate market cap. A/B Vendors: sorted by alphabetical order
Ronny Kohavi was an executive at Amazon, Microsoft, and Airbnb and has over 20 years of experience running A/B tests and leading experimentation teams. He loves to teach, and his papers have over 55,000 citations. He co-authored the best-selling book: Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing (with Diane Tang and Ya Xu), which is a top-10 data mining book on Amazon. He is the most viewed writer on Quora's A/B testing and received the Individual Lifetime Achievement Award for Experimentation Culture in Sept 2020.
Ronny holds a PhD in Machine Learning from Stanford University.
See more at http://www.kohavi.com
10 hours total over 5 x 2-hour sessions
Week 1: Mon, Tue, Thu
8-10AM Pacific Time
Three x 2-hour sessions in week 1 on Monday, Tuesday, and Thursday
Week 2: Mon, Thu
8-10AM Pacific Time
Two x 2-hour sessions in week 2 on Monday and Thursday
Optional Q&A
15 minutes after each session
A/B testing book Chapter 1
Interested in reading chapter 1 of my book: Trustworthy Online Controlled Experiments : A Practical Guide to A/B Testing?
Get it by entering your email
Join an upcoming cohort
2 Dec 2024
$1,999
Dates
Payment Deadline
Real-world examples
We will review multiple real A/B tests
Deep dive design and analysis of two A/B tests
We will deep dive into the full lifecycle of designing an A/B test to answer a hypothesis and analyze the results
Learn with a cohort of peers
Join a community of like-minded people who want to learn and grow alongside you
Requested topics
Missing anything? We have allocated time to suggest topics, collect votes, and discuss them on the last session
Can I get reimbursed by my employer?
What happens if I can’t make a live session?
I work full-time, what is the expected time commitment?
Are there small company group discounts?
Are there discounts for large groups?
Can I pay through a bank transfer / ACH?
What's the refund policy?
Sign up to be the first to know about course updates.
Join an upcoming cohort
2 Dec 2024
$1,999
Dates
Payment Deadline