4.8 (3)
Cohort-based Course
Learn from a world-leading expert the essentials of accelerating innovation with trustworthy A/B tests - private class
4.8 (3)
Cohort-based Course
Learn from a world-leading expert the essentials of accelerating innovation with trustworthy A/B tests - private class
Course overview
NOTE: this is designed for private classes of at least 40 people. You pick the dates. Sign-up for the waitlist and reply to the email if you are interested in scheduling a class.
This is about 6 hours of content taught over an 8-hour day (with breaks).
The costs are detailed in https://bit.ly/ABCourseBulkDiscounts under "Essentials" class.
For the flagship live class, visit https://bit.ly/ABClassRKFlagship (Maven doesn't allow making this link clickable, so please copy-paste).
---
Through multiple real examples of well-run experiments and real stories at Microsoft, Amazon, and Airbnb, you will see the humbling reality that we are terrible at assessing the values of ideas.
Trivial changes can be surprisingly useful, whereas large efforts often fail. A/B testing allows you to evaluate changes and accelerate innovation by building Minimum Viable Products and Features (MVPs). The organization becomes evidence-based and humbler, as it adopts and learns to use evidence from the gold standard in science: the controlled experiment.
You will understand the challenges in designing and running trustworthy controlled experiments, or A/B tests, including the importance of the Overall Evaluation Criterion (OEC), scaling, pitfalls, and Twyman's law.
Learn the essentials of A/B testing by watching easy-to-consume videos at your own pace, ask questions on the Maven questions channel, and attend two interactive one-hour sessions for fun polls and Q&A.
01
Organization leaders (VPs, directors, managers) interested in understanding the essentials of A/B testing to accelerate innovation
02
Program managers focused on growth, revenue, conversions, and prioritization who will understand how to better design robust clear metrics
03
Data science managers and scientists who do not need or want the 2nd part of https://bit.ly/ABClassRKFlagship (or prefer videos)
04
Anyone interested in making the organizations more data-driven and efficient with fewer severe incidents
Understand and internalize the humbling reality that we are poor at assessing the values of ideas: most ideas fail!
You will hear multiple real memorable stories and examples, many that don't make it to books or articles. These were chosen from over 20 years of experimentation. You'll have the data to show that the poor success rate is documented across multiple organizations; expected it!
Understand the key advantages and limitations of A/B testing
Understand key concepts like causality, hierarchy of evidence, and key organizational tenets required for effective experimentation.
Learn how to design metrics and the Overall Evaluation Criterion
Designing metrics is hard. There is a hierarchy of metrics and perverse incentives. The most important metrics comprise of the OEC - The Overall Evaluation Criterion. We will look at good and bad examples.
Learn how to designing trustworthy A/B tests
Getting numbers is easy; getting numbers you can trust is hard. You'll understand common pitfalls and how to design reliable and trustworthy tests.
Learn how to generate and prioritize ideas
You'll understand the key tradeoff between the cost and fidelity of different methods. You'll learn about the key concept of EVI: the Expected Value of Information. You'll learn about different prioritization frameworks.
Interactive live sessions
Lifetime access to course materials
17 in-depth lessons
Direct access to instructor
Projects to apply learnings
Guided feedback & reflection
Private community of peers
Course certificate upon completion
Innovation & AB Testing - the Essentials
4.8 (3)
Dylan Lewis
Ryan Lucht
Pavan Gangisetty
Han Dong
Ishan Goel
Deborah O'Malley
Jialin Huang
Scott Rome
Aaron Gasperi
Manuel de Francisco Vera
Top: Companies sorted by approximate market value. A/B Vendors: sorted by alphabetical order
Ronny Kohavi was an executive at Amazon, Microsoft, and Airbnb and has over 20 years of experience running A/B tests and leading experimentation teams. He loves to teach, and his papers have over 60,000 citations. He co-authored the best-selling book: Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing (with Diane Tang and Ya Xu), which is a top-10 data mining book on Amazon. He is the most viewed writer on Quora's A/B testing and received the Individual Lifetime Achievement Award for Experimentation Culture in Sept 2020.
Ronny holds a PhD in Machine Learning from Stanford University.
See more at http://www.kohavi.com
One full day: 6 hours of content+breaks
6 hours of content
We jointly determine structure and break
A/B testing book Chapter 1
Interested in reading chapter 1 of my book: Trustworthy Online Controlled Experiments : A Practical Guide to A/B Testing?
Get this free resource
Be the first to know about upcoming cohorts
Real-world examples
We will review multiple real A/B tests
Deep dive design and analysis of an A/B test - end to end
We will deep dive into the full lifecycle of designing an A/B test to answer a hypothesis and analyze the results
Learn with a cohort of peers
Join a community of like-minded people who want to learn and grow alongside you
Be the first to know about upcoming cohorts