3 March 2025

Learn from a world-leading expert how to design and analyze trustworthy A/B tests to evaluate ideas, integrate AI/ML, and grow your business

Dr. Ronny Kohavi

Accelerating Innovation with AB Testing

Microsoft

Airbnb

Amazon

Through multiple real examples of well-run experiments and real stories at Microsoft, Amazon, and Airbnb, you will see the humbling reality that we are terrible at assessing the values of ideas. Trivial changes can be surprisingly useful, whereas large efforts often fail. Accelerate innovation by building Minimum Viable Products and Features (MVPs) and make the organization evidence-based and humbler, as it adopts and learns to use evidence from the gold standard in science: the controlled experiment. You will understand the challenges in designing and running trustworthy controlled experiments, or A/B tests, including the importance of the Overall Evaluation Criterion (OEC), scaling, pitfalls, and Twyman's law. The 2nd week covers additional topics, some more technical, including cultural challenges, institutional memory, maturity model, observational causal studies, offline evaluations, AI/Machine learning and triggering, scaling, build vs. buy, challenges, and requested topics. The advanced topics course is a follow-on to this: https://bit.ly/AdvancedABFS

Empower your organization to be data-driven and innovative

Data science managers and scientists will be able to design and interpret the experiment results in a trustworthy manner

Program managers focused on growth, revenue, conversions, and prioritization will understand how to provide the org with robust clear metric

Engineering leaders will be able to make the organizations more data-driven and efficient with fewer severe incidents through A/B tests

Who is this class for?

What you will learn from this 10-hour course

You will hear multiple real memorable stories and examples, many that don't make it to books or articles. These were chosen from over 20 years of experimentation. You'll have the data to show that the poor success rate is documented across multiple organizations; expected it!

Understand and internalize the humbling reality that we are poor at assessing the values of ideas: most ideas fail!

Understand key concepts like causality, hierarchy of evidence, and key organizational tenets required for effective experimentation.

Understand the key advantages and limitations of A/B testing

Designing metrics is hard. There is a hierarchy of metrics and perverse incentives. The most important metrics comprise of the OEC - The Overall Evaluation Criterion. We will look at good and bad examples.

Learn how to design metrics and the Overall Evaluation Criterion

Getting numbers is easy; getting numbers you can trust is hard. You'll understand common pitfalls and how to design reliable and trustworthy tests.

Learn how to designing trustworthy A/B tests

Learn about the cultural challenges, the humbling results (most ideas fail, pivoting, iterating, learning), institutional memory, ideation, prioritization, experimentation platforms

Learn about the cultural challenges

When building AI or machine learning models, using A/B testing and triggering to evaluate the models that were built offline based on historical data

Learn about the relationship to AI and Machine Learning Modeling

When you can't run an A/B test, quasi-experimentation methods and the risks of observational causal studies

Learn about complementary quasi-experimental techniques

What are key challenges and open questions in the field

Learn about existing challenges

If there is something specific you want to cover, there is time allocated for topics voted by the audience to discuss

Learn about YOUR topic of interest

The course focuses on developing the intuition and common misunderstandings, without the details of the statistics, which you can find in many books. We cover p-values, statistical power, and triggering.You can go as technical as you want in the Q&amp;A and community discussions

Technical to your needs

<ul><li>Introduction to controlled experiments, or A/B testing</li><li>Interesting experiments – you’re the decision maker</li><li>Organizational tenets</li><li>Advantages and limitations of controlled experiments</li></ul>

Introduction

<ul><li>Interesting example</li><li>End-to-end experiment: Running and Analyzing an Experiment</li><li>Metrics</li><li>The OEC</li></ul>

End-to-end example, and Metrics and the OEC

<ul><li>Interesting example</li><li>P-values and statistical power</li><li>End-to-end experiment: speed matters</li><li>Twyman’s Law and Trustworthy Experimentation</li><li>Generating ideas, prioritization, ethics</li></ul>

P-values and power, End-to-end Example 2, Twyman's law and Trustworthy Experimentation

<ul><li>Interesting example</li><li>Cultural challenge, Institutional memory, maturity model</li><li>Observational causal studies and quasi-experimental designs</li><li>Pitfalls in observational studies</li></ul>

Cultural challenges, Prioritization, AI/Machine Learning, Complementary Techniques and Observational Causal Studies

<ul><li>AI/Machine Learning model, and triggering</li><li>Leakage and interference</li><li>Bayesian A/B Testing</li><li>Experimentation Platform and scaling</li><li>Build vs. Buy</li><li>Challenges</li><li>Requested Topics</li></ul>

Advanced topics, challenges, and requested topics

Course syllabus

100% Customer Satisfaction Guarantee

Sr. Business Intelligence Engineer @ Credit Karma

Experimentation Product Manager @ Ford Credit

Director of Experimentation @ Kaiser Permanente

Chief editor of GoodUI - Conversion focused UI Designer

Founder & CEO @ GuessTheTest, named a "top-10" digital marketing experts to follow

Head of Experimentation, Personalization, Marketing, and User Understanding at Zillow

Director, Customer Knowledge & Strategic Insights @ Fidelity

Staff Software Development Engineer, OfferUp

Sr. Director, Data Science & Analytics, Vinted

Conversion Optimization Specialist, Bitdefender

Testimonials: What People on Sphere wrote (the platform where most cohorts were taught before Maven)

Companies with two or more people
that took the course 

Meet your instructor

Three x 2-hour sessions in week 1 on Monday, Tuesday, and Thursday

Week 1: Mon, Tue, Thu

Two x 2-hour sessions in week 2 on Monday and Thursday

Week 2: Mon, Thu

Optional Q&A

Course schedule

We will review multiple real A/B tests

Real-world examples

We will deep dive into the full lifecycle of designing an A/B test to answer a hypothesis and analyze the results

Deep dive design and analysis of two A/B tests

Join a community of like-minded people who want to learn and grow alongside you

Learn with a cohort of peers

Missing anything? We have allocated time to suggest topics, collect votes, and discuss them on the last session

Course completed by

About this course

Get notified about the next cohort

Learn from live sessions