Hero image

Accelerating Innovation with AB Testing

9.0

(34 ratings)

·

2 Weeks

·

Cohort-based Course

Learn from a world-leading expert how to design and analyze trustworthy A/B tests to evaluate ideas, integrate AI/ML, and grow your business

Hosted by

Dr. Ronny Kohavi

Technical fellow and VP at Microsoft & Airbnb, co-author of A/B testing book

This course is popular

6 people enrolled last week.

Course overview

Learn the humbling reality: we are terrible at assessing the values of ideas

Through multiple real examples and real stories, you will see the humbling reality of surprising experiments that ran at companies including Microsoft, Amazon, and Airbnb.


You will understand the cultural and technical challenges in designing and running trustworthy controlled experiments, or A/B tests, including the importance of the Overall Evaluation Criterion (OEC), scaling, pitfalls, Twyman's law, triggering, ethics, and many unaddressed challenges.

Designed to help make organizations more data-driven

01

Data science managers and scientists will be able to design and interpret the experiment results in a trustworthy manner

02

Program managers focused on growth, revenue, conversions, and prioritization will understand how to provide the org with robust clear metric

03

Engineering leaders will be able to make the organizations more data-driven and efficient with fewer severe incidents through A/B tests

What you’ll get out of this 10-hour course

Real examples and real stories

The advantage of a live course is the benefit of hearing the real memorable stories, many that don't make it to books or articles, chosen from over 20 years of experimentation: motivation, pushback, success/failure rates

Introduction to A/B Testing

Understand the key benefits of controlled experiments, or A/B tests, including causality, surprising examples, metrics, interpreting results, trust and pitfalls, Twyman's law, and A/A tests

How to design metrics and the Overall Evaluation Criterion

Designing metrics is hard. There is a hierarchy of metrics and perverse incentives. The most important metrics comprise of the OEC - The Overall Evaluation Criterion. We will look at good and bad examples.

Getting numbers you can trust is hard

Getting numbers is easy; getting numbers you can trust is hard. We'll discuss pitfalls and how to build a reliable and trustworthy experimentation system.

Cultural challenges and the organization

Learn about the cultural challenges, the humbling results (most ideas fail, pivoting, iterating, learning), institutional memory, ideation, prioritization, experimentation platforms

AI and Machine Learning Modeling

When building AI or machine learning models, using A/B testing and triggering to evaluate the models that were built offline based on historical data

Technical to your needs

The course focuses on developing the intuition and common misunderstandings, without the details of the statistics, which you can find in many books. We cover p-values, statistical power, and triggering.

You can go as technical as you want in the Q&A and community discussions

Complementary techniques

When you can't run an A/B test, quasi-experimentation methods and the risks of observational causal studies

Challenges

What are key challenges in the field

Requested Topics

If there is something specific you want to cover, there is time allocated for that

100% Customer Satisfaction Guarantee

Not satisfied after attending the first session and before session 2 starts? Get a full refund
Not satisfied after attending the first session and before session 2 starts? Get a full refund

9.0

(34 ratings)

Testimonials: What People Said

        Ronny made the content amazingly easy to consume...I can't recommend it enough
Dylan Lewis

Dylan Lewis

Experimentation Leader at Atlassian
        If you're interested in running A/B tests and learning the science behind them, my highest recommendation is to take Ronny Kohavi's cohort-based course on the topic. (The book Ronny co-authored is tied for first on my reading list too.)
Ryan Lucht

Ryan Lucht

Director of Growth Strategy, Cro Metrics
        The entire course and the way it was delivered was amazing - had tons of learning, especially on where we could go wrong. Examples followed by key concepts was great. Really appreciate Ronny taking time to answer each question even after the session.
Pavan Gangisetty

Pavan Gangisetty

Staff Data Analyst @ Intuit
        The culture of Q&A during the session. The depth and width of the experimentation topic. Really eye-opening learnings from Ronny. Thank you, thank you, thank you!
Han Dong

Han Dong

Sr. Business Intelligence Engineer @ Credit Karma
        Ronny covered similar topics that are in his book, but in a way that internalizes the points much deeper.
Scott Theisen

Scott Theisen

Experimentation Product Manager @ Ford Credit
        I love that the session goes beyond the tech & stats of experimentation and also address the cultural aspects.
Sharath Bulusu

Sharath Bulusu

Director of Product Management @ Google
        The curation of relevant examples/stories provides the type of evidence often needed to make a case internally. So helpful to have it assembled in a clear strategy/direction.
James Niehaus

James Niehaus

Director of Experimentation @ Kaiser Permanente
        I love Ronny's stories. I think that apart from the learning, two of the most valuable things from this class are: 1) Stories because you will remember them and they will come in handy and 2) Links so that you can dive deeper.
Ishan Goel

Ishan Goel

Lead Data Scientist @ Wingify
        Ronny is the authority in experimentation...I highly recommend Ronny as a teacher in all things experimentation as I learned a ton from him while working with Microsoft. Ronny is extremely skilled in communicating deep technical knowledge with clarity, friendliness and expertise.
Jakub Linowski

Jakub Linowski

Chief editor of GoodUI - Conversion focused UI Designer
        50% of what I've learned myself about experimentation is wrong. The other half I learned from Ronny.
Deborah O'Malley

Deborah O'Malley

Founder & CEO @ GuessTheTest, named a "top-10" digital marketing experts to follow
        Want to take on the challenge of evolving the scientific culture in your organization with experimentation? I'd highly recommend considering Ronny Kohavi's Sphere cohort...lessons from Ronny's career driving experimentation culture and impact, and invaluable networking opportunities with experimentation leaders.
Aaro Wroblewski

Aaro Wroblewski

Head of Experimentation, Personalization, Marketing, and User Understanding at Zillow
        Ronny's tons of experience is valuable and hearing it firsthand is amazing.
Jialin Huang

Jialin Huang

Director, Customer Knowledge & Strategic Insights @ Fidelity
        Favorite part: Practical info and learnings that weren't in the book but were probably not easily publishable.
Scott Rome

Scott Rome

Senior Principal Engineer @ Comcast
        This is an amazing opportunity. I think of all of the times I have questioned, “does X make sense?” I saved up all of those questions. I’ll tell you that Ronny Kohavi covered MANY of them in his lecture content (so I didn’t have to ask those). It was so powerful to just ask directly. Easily saved me hundreds if not thousands of hours of research.
Aaron Gasperi

Aaron Gasperi

Principal product manager, Walmart
        I had been running a/b tests for many years early on in my career thinking I was doing it "right", and Ronny's class opened my eyes to common mistakes I was making that could completely invalidate my learnings...I often find myself referencing best practices and methodologies learned from Ronny's course in my every day interactions.
Jessica Porges

Jessica Porges

Senior Experimentation Advisor, Split
        An exceptional course that provided in-depth discussions on real-world examples. The class poll, which tested real ideas implemented by big companies, and the winner after A/B testing was an unforgettable experience that cannot be obtained through book reading alone… I find myself going back to the course slides for reference, like today.
Haiyan Chen

Haiyan Chen

Staff Software Development Engineer, OfferUp
        I recently completed Ronny Kohavi's "Accelerating Innovation" program, and it was a truly mind-blowing experience. Throughout the course, I gained practical knowledge and hands-on experience in A/B testing, as well as online and offline testing of machine learning models, interleaving, and causal inference methods....
Emma Ding

Emma Ding

Data Science Career Coach
        Just had an incredible experience as the entire leadership team at Vinted Marketplace joined me in the 'Accelerating Innovation with AB Testing' course by Ronny Kohavi himself! 💪🔥 …Let's take our experimentation game to the next level! 
Manuel de Francisco Vera

Manuel de Francisco Vera

Sr. Director, Data Science & Analytics, Vinted
        Great real examples - love it!
Markus Wiggering

Markus Wiggering

Head of Product Management, CHECK24
        Loved all of it <3 Ron answered all the questions and the courses were exactly what I was looking for.
Sorin Tarna

Sorin Tarna

Conversion Optimization Specialist, Bitdefender
        This course is awesome! I think it is applicable for those with more-matured experimentation cultures and for smaller, scrappier teams navigating the dichotomy of product development and finding PMF. Great insights across the board and Ronny is a great explainer!
Gabriel Rodriguez

Gabriel Rodriguez

Senior Product Manager, Dr. Squatch

Companies with two or more people that took the course

Top: Companies sorted by approximate market value.
A/B Vendors: sorted by alphabetical order
Top: Companies sorted by approximate market value. A/B Vendors: sorted by alphabetical order

Meet your instructor

Dr. Ronny Kohavi

Dr. Ronny Kohavi

Ronny Kohavi was an executive at Amazon, Microsoft, and Airbnb and has over 20 years of experience running A/B tests and leading experimentation teams. He loves to teach, and his papers have over 55,000 citations. He co-authored the best-selling book: Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing (with Diane Tang and Ya Xu), which is a top-10 data mining book on Amazon. He is the most viewed writer on Quora's A/B testing and received the Individual Lifetime Achievement Award for Experimentation Culture in Sept 2020.


Ronny holds a PhD in Machine Learning from Stanford University.


See more at http://www.kohavi.com

Course syllabus

01

Introduction

  • Introduction to controlled experiments, or A/B testing
  • Interesting experiments – you’re the decision maker
  • Organizational tenets
  • Advantages and limitations of controlled experiments

02

End-to-end example, and Metrics and the OEC

  • Interesting example
  • End-to-end experiment: Running and Analyzing an Experiment
  • Metrics
  • The OEC

03

P-values and power, End-to-end Example 2, Twyman's law and Trustworthy Experimentation

  • Interesting example
  • P-values and statistical power
  • End-to-end experiment: speed matters
  • Twyman’s Law and Trustworthy Experimentation
  • Generating ideas, prioritization, ethics

04

Cultural challenges, Prioritization, AI/Machine Learning, Complementary Techniques and Observational Causal Studies

  • Interesting example
  • Cultural challenge, Institutional memory, maturity model
  • Observational causal studies and quasi-experimental designs
  • Pitfalls in observational studies

05

Advanced topics, challenges, and requested topics

  • AI/Machine Learning model, and triggering
  • Leakage and interference
  • Bayesian A/B Testing
  • Experimentation Platform and scaling
  • Build vs. Buy
  • Challenges
  • Requested Topics

Course schedule

10 hours total over 5 x 2-hour sessions
  • Week 1: Mon, Tue, Thu

    8-10AM Pacific Time

    Three x 2-hour sessions in week 1 on Monday, Tuesday, and Thursday

  • Week 2: Mon, Thu

    8-10AM Pacific Time

    Two x 2-hour sessions in week 2 on Monday and Thursday


  • Optional Q&A

    15 minutes after each session


Free resource

A/B testing book Chapter 1

Interested in reading chapter 1 of my book: Trustworthy Online Controlled Experiments : A Practical Guide to A/B Testing?


Get it by entering your email
A pattern of wavy dots
Join an upcoming cohort

Accelerating Innovation with AB Testing

4 Dec 2023

$1,999 USD

Dates

Dec 4—14, 2023

Payment Deadline

Dec 4, 2023

Learning is better with cohorts

Learning is better with cohorts

Real-world examples

We will review multiple real A/B tests

Deep dive design and analysis of two A/B tests

We will deep dive into the full lifecycle of designing an A/B test to answer a hypothesis and analyze the results

Learn with a cohort of peers

Join a community of like-minded people who want to learn and grow alongside you

Requested topics

Missing anything? We have allocated time to suggest topics, collect votes, and discuss them on the last session

Frequently Asked Questions

Can I get reimbursed by my employer?
What happens if I can’t make a live session?
I work full-time, what is the expected time commitment?
Are there company group discounts?
Can I pay through a bank transfer / ACH?
What's the refund policy?
Is there a cheaper/shorter course?

Stay in the loop

Sign up to be the first to know about course updates.

A pattern of wavy dots
Join an upcoming cohort

Accelerating Innovation with AB Testing

4 Dec 2023

$1,999 USD

Dates

Dec 4—14, 2023

Payment Deadline

Dec 4, 2023

$1,999 USD

9.0

(34)

·

2 Weeks