Nov 2023

Learn from a world-leading expert the essentials of accelerating innovation with trustworthy A/B tests - private class

Dr. Ronny Kohavi

Innovation & AB Testing - the Essentials

NOTE: this is designed for private classes of at least 40 people. You pick the dates. Sign-up for the waitlist and reply to the email if you are interested in scheduling a class. This is about 6 hours of content taught over an 8-hour day (with breaks). The costs are detailed in https://bit.ly/ABCourseBulkDiscounts under "Essentials" class. For the flagship live class, visit https://bit.ly/ABClassRKFlagship (Maven doesn't allow making this link clickable, so please copy-paste).--- Through multiple real examples of well-run experiments and real stories at Microsoft, Amazon, and Airbnb, you will see the humbling reality that we are terrible at assessing the values of ideas. Trivial changes can be surprisingly useful, whereas large efforts often fail. A/B testing allows you to evaluate changes and accelerate innovation by building Minimum Viable Products and Features (MVPs). The organization becomes evidence-based and humbler, as it adopts and learns to use evidence from the gold standard in science: the controlled experiment. You will understand the challenges in designing and running trustworthy controlled experiments, or A/B tests, including the importance of the Overall Evaluation Criterion (OEC), scaling, pitfalls, and Twyman's law. Learn the essentials of A/B testing by watching easy-to-consume videos at your own pace, ask questions on the Maven questions channel, and attend two interactive one-hour sessions for fun polls and Q&amp;A.

Empower your organization to be data-driven and innovative

Organization leaders (VPs, directors, managers) interested in understanding the essentials of A/B testing to accelerate innovation

Program managers focused on growth, revenue, conversions, and prioritization who will understand how to better design robust clear metrics

Data science managers and scientists who do not need or want the 2nd part of https://bit.ly/ABClassRKFlagship (or prefer videos)

Anyone interested in making the organizations more data-driven and efficient with fewer severe incidents

Who is this Course For?

What you will learn from this course

You will hear multiple real memorable stories and examples, many that don't make it to books or articles. These were chosen from over 20 years of experimentation. You'll have the data to show that the poor success rate is documented across multiple organizations; expected it!

Understand and internalize the humbling reality that we are poor at assessing the values of ideas: most ideas fail!

Understand key concepts like causality, hierarchy of evidence, and key organizational tenets required for effective experimentation.

Understand the key advantages and limitations of A/B testing

Designing metrics is hard. There is a hierarchy of metrics and perverse incentives. The most important metrics comprise of the OEC - The Overall Evaluation Criterion. We will look at good and bad examples.

Learn how to design metrics and the Overall Evaluation Criterion

Getting numbers is easy; getting numbers you can trust is hard. You'll understand common pitfalls and how to design reliable and trustworthy tests.

Learn how to designing trustworthy A/B tests

You'll understand the key tradeoff between the cost and fidelity of different methods. You'll learn about the key concept of EVI: the Expected Value of Information. You'll learn about different prioritization frameworks.

Learn how to generate and prioritize ideas

Videos (view on your own asynchronously):<ul><li>Introduction to controlled experiments, or A/B tests</li><li>Course overview and housekeeping</li><li>Observational studies and the hierarchy of evidence</li></ul>

Videos: Introduction and Motivation

Videos (view on your own asynchronously):<ul><li>Tenets &amp; introduction to the OEC</li><li>Advantages &amp; issues</li><li>Safe deployments</li></ul>

Videos: A/B Testing: tenets, advantage & issues

<ul><li>Metrics Taxonomy</li><li>Gameability and Goodhart's law</li><li>OEC parts 1,2</li></ul>

Videos: Metrics Taxonomy & the OEC

<ul><li>Interesting A/B tests</li><li>Surprising experiments</li><li>Key Lessons</li><li>Q&amp;A</li></ul>

Monday: Live session

<ul><li>End-to-end example: coupon codes</li><li>What's a p-value?</li><li>Statistical power</li></ul>

Videos: Designing and Interpreting Experiments

<ul><li>Twyman's law and trust</li><li>Generating Ideas, prioritization and ethics</li></ul>

Videos: Twyman's Law, Trust, and idea generation, Prioritization, and Ethics

<ul><li>Additional Topics</li><li>What's next</li></ul>

Videos:
 

<ul><li>A/B test polls</li><li>Requested Topics</li><li>Q&amp;A</li></ul>

Thursday: Live session

Course syllabus

100% Customer Satisfaction Guarantee

Sr. Business Intelligence Engineer @ Credit Karma

Founder & CEO @ GuessTheTest, named a "top-10" digital marketing experts to follow

Director, Customer Knowledge & Strategic Insights @ Fidelity

Sr. Director, Data Science & Analytics, Vinted

Attendee Testimonials: What People Said  
(from the longer course)

Companies with two or more people
that took my courses

Meet your instructor

6 hours of content

Course schedule

We will review multiple real A/B tests

Real-world examples

We will deep dive into the full lifecycle of designing an A/B test to answer a hypothesis and analyze the results

Deep dive design and analysis of an A/B test - end to end

Join a community of like-minded people who want to learn and grow alongside you

Course completed by

About this course

Get notified about the next cohort

Learn from live sessions