Hero image

Innovation & AB Testing - the Essentials

New
·

2 Weeks

·

Cohort-based Course

Learn from videos and live sessions with a world-leading expert how to design and analyze trustworthy A/B tests to accelerate innovation

Hosted by

Dr. Ronny Kohavi

Technical fellow and VP at Microsoft & Airbnb, co-author of A/B testing book

Course overview

Learn the humbling reality: we are terrible at assessing the values of ideas

Through multiple real examples and real stories, you will see the humbling reality of surprising experiments that ran at companies including Microsoft, Amazon, and Airbnb.


You will understand the challenges in designing and running trustworthy controlled experiments, or A/B tests, including the importance of the Overall Evaluation Criterion (OEC), scaling, pitfalls, and Twyman's law.


This is a shorter and less technical version of the flagship course https://bit.ly/ABClassRKFlagship. You will view videos at your own pace and attend two interactive one-hour sessions.

Designed to help make organizations more data-driven

01

Anyone interested in A/B testing that is too busy to take the flagship course https://bit.ly/ABClassRKFlagship, or prefers async learning

02

Engineering leaders will be able to make the organizations more data-driven and efficient with fewer severe incidents

03

Program managers focused on growth, revenue, conversions, and prioritization will understand how to design robust clear metrics

04

Data science managers and scientists who do not need or want the technical details, but rather looking for the high-level overview

What you’ll get out of this course

Real examples and real stories

You will hear multiple real memorable stories and examples, many that don't make it to books or articles. These were chosen from over 20 years of experimentation: motivation, pushback, success/failure rates

Introduction to A/B Testing

Understand the key benefits of controlled experiments, or A/B tests, including causality, hierarchy of evidence, surprising examples, metrics, interpreting results, trust and pitfalls, Twyman's law, and A/A tests

How to design metrics and the Overall Evaluation Criterion

Designing metrics is hard. There is a hierarchy of metrics and perverse incentives. The most important metrics comprise of the OEC - The Overall Evaluation Criterion. We will look at good and bad examples.


Getting numbers you can trust is hard

Getting numbers is easy; getting numbers you can trust is hard. We'll discuss pitfalls and how to build a reliable and trustworthy experimentation system.


100% Customer Satisfaction Guarantee

Not satisfied after seeing some videos? Get a full refund within 48 hours of the course start
Not satisfied after seeing some videos? Get a full refund within 48 hours of the course start

Attendee Testimonials: What People Said (from the flagship course)

        Ronny made the content amazingly easy to consume...I can't recommend it enough
Dylan Lewis

Dylan Lewis

Experimentation Leader at Atlassian
        If you're interested in running A/B tests and learning the science behind them, my highest recommendation is to take Ronny Kohavi's cohort-based course on the topic. (The book Ronny co-authored is tied for first on my reading list too.)
Ryan Lucht

Ryan Lucht

Director of Growth Strategy, Cro Metrics
        The entire course and the way it was delivered was amazing - had tons of learning, especially on where we could go wrong. Examples followed by key concepts was great. Really appreciate Ronny taking time to answer each question even after the session.
Pavan Gangisetty

Pavan Gangisetty

Staff Data Analyst @ Intuit
        The culture of Q&A during the session. The depth and width of the experimentation topic. Really eye-opening learnings from Ronny. Thank you, thank you, thank you!
Han Dong

Han Dong

Sr. Business Intelligence Engineer @ Credit Karma
        I love Ronny's stories. I think that apart from the learning, two of the most valuable things from this class are: 1) Stories because you will remember them and they will come in handy and 2) Links so that you can dive deeper.
Ishan Goel

Ishan Goel

Lead Data Scientist @ Wingify
        50% of what I've learned myself about experimentation is wrong. The other half I learned from Ronny.
Deborah O'Malley

Deborah O'Malley

Founder & CEO @ GuessTheTest, named a "top-10" digital marketing experts to follow
        Ronny's tons of experience is valuable and hearing it firsthand is amazing.
Jialin Huang

Jialin Huang

Director, Customer Knowledge & Strategic Insights @ Fidelity
        Favorite part: Practical info and learnings that weren't in the book but were probably not easily publishable.
Scott Rome

Scott Rome

Senior Principal Engineer @ Comcast
        This is an amazing opportunity. I think of all of the times I have questioned, “does X make sense?” I saved up all of those questions. I’ll tell you that Ronny Kohavi covered MANY of them in his lecture content (so I didn’t have to ask those). It was so powerful to just ask directly. Easily saved me hundreds if not thousands of hours of research.
Aaron Gasperi

Aaron Gasperi

Principal product manager, Walmart
        Just had an incredible experience as the entire leadership team at Vinted Marketplace joined me in the 'Accelerating Innovation with AB Testing' course by Ronny Kohavi himself! 💪🔥 …Let's take our experimentation game to the next level! 
Manuel de Francisco Vera

Manuel de Francisco Vera

Sr. Director, Data Science & Analytics, Vinted

Companies with two or more people that took the flagship course

Top: Companies sorted by approximate market value. A/B Vendors: sorted by alphabetical order
Top: Companies sorted by approximate market value. A/B Vendors: sorted by alphabetical order

Meet your instructor

Dr. Ronny Kohavi

Dr. Ronny Kohavi

Ronny Kohavi was an executive at Amazon, Microsoft, and Airbnb and has over 20 years of experience running A/B tests and leading experimentation teams. He loves to teach, and his papers have over 60,000 citations. He co-authored the best-selling book: Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing (with Diane Tang and Ya Xu), which is a top-10 data mining book on Amazon. He is the most viewed writer on Quora's A/B testing and received the Individual Lifetime Achievement Award for Experimentation Culture in Sept 2020.


Ronny holds a PhD in Machine Learning from Stanford University.


See more at http://www.kohavi.com

Course syllabus

01

Videos: Introduction and Motivation

Videos (view on your own asynchronously):

  • Introduction to controlled experiments, or A/B tests
  • Motivation: observational studies and the hierarchy of evidence

02

Videos: A/B Testing: tenets, advantage & issues, example

Videos (view on your own asynchronously):

  • Tenets & introduction to the OEC
  • Advantages & issues
  • Safe deployments

03

Videos: Metrics Taxonomy & the OEC

  • Metrics Taxonomy
  • Gameability and Goodhart's law
  • OEC part 1,2

04

Monday: Live session

  • Interesting A/B tests
  • Q&A


05

Videos: Designing and Interpreting Experiments

  • End-to-end example: coupon codes
  • What's a p-value?
  • Statistical power


06

Videos: Twyman's Law, Trust, and idea generation and prioritization

  • Twyman's law and trust
  • Generating Ideas and Prioritization



07

Videos: Additional Topics

  • Additional Topics
  • What's next


08

Thursday: Live session

  • A/B test polls
  • Requested Topics
  • Q&A


Course schedule

4 hours of videos + 2 hours live

  • Week 1: watch 2.5 hours of videos

    Watch videos on your own


    • Ask questions on community forum channel

  • Week 2: two live sessions, 1.5 hours of video

    Attend live session at 8AM or 5PM PST

    • Live session Monday - choose either 8AM or 5PM PST. 8AM PST works well for the Americas, Europe, and late in India. 5PM works well for Australia, New Zealand, Japan, Americas west coast, and early in India
    • Watch 1.5 hours of video on Tuesday/Wednesday
    • Live session Thursday
Free resource

A/B testing book Chapter 1

Interested in reading chapter 1 of my book: Trustworthy Online Controlled Experiments : A Practical Guide to A/B Testing?

Get this free resource
A pattern of wavy dots
Join an upcoming cohort

Innovation & AB Testing - the Essentials

Nov 2023

$999 USD

Dates

Nov 6—17, 2023

Payment Deadline

Nov 6, 2023

Get reimbursed by your employer

Learning is better with cohorts

Learning is better with cohorts

Real-world examples

We will review multiple real A/B tests

Deep dive design and analysis of an A/B test - end to end

We will deep dive into the full lifecycle of designing an A/B test to answer a hypothesis and analyze the results

Learn with a cohort of peers

Join a community of like-minded people who want to learn and grow alongside you

Requested topics

Missing anything? Suggest topics, vote, and we will discuss in the last live session

Frequently Asked Questions

What happens if I can’t make a live session?
Are there discounts for X?
Can I pay through a bank transfer / ACH?
What's the refund policy?
Can I continue learning and attend the 2nd week of the longer flagship course?
A pattern of wavy dots
Join an upcoming cohort

Innovation & AB Testing - the Essentials

Nov 2023

$999 USD

Dates

Nov 6—17, 2023

Payment Deadline

Nov 6, 2023

Get reimbursed by your employer

$999 USD

New
·

2 Weeks

Powered by Maven