Innovation & AB Testing - the Essentials

4.8

(3 ratings)

2 Weeks

Cohort-based Course

Learn from a world-leading expert the essentials of accelerating innovation with trustworthy A/B tests. Recorded videos and live sessions.

Innovation & AB Testing - the Essentials

4.8

(3 ratings)

2 Weeks

Cohort-based Course

Learn from a world-leading expert the essentials of accelerating innovation with trustworthy A/B tests. Recorded videos and live sessions.

Hosted by

Dr. Ronny Kohavi

Technical fellow and VP at Microsoft & Airbnb, co-author of A/B testing book

Dr. Ronny Kohavi

Technical fellow and VP at Microsoft & Airbnb, co-author of A/B testing book

Course overview

Empower your organization to be data-driven and innovative

NOTE: this video with live sessions course is designed for private (at least 30 people) or semi-private ( at least 20 people from your company; you pick the dates, others can join). Sign-up for the waitlist and reply to the email if you are interested in scheduling a class.

---

Through multiple real examples of well-run experiments and real stories at Microsoft, Amazon, and Airbnb, you will see the humbling reality that we are terrible at assessing the values of ideas.

Trivial changes can be surprisingly useful, whereas large efforts often fail. A/B testing allows you to evaluate changes and accelerate innovation by building Minimum Viable Products and Features (MVPs). The organization becomes evidence-based and humbler, as it adopts and learns to use evidence from the gold standard in science: the controlled experiment.

You will understand the challenges in designing and running trustworthy controlled experiments, or A/B tests, including the importance of the Overall Evaluation Criterion (OEC), scaling, pitfalls, and Twyman's law.

Learn the essentials of A/B testing by watching easy-to-consume videos at your own pace, ask questions on the Maven questions channel, and attend two interactive one-hour sessions for fun polls and Q&A.

Who is this Course For?

Organization leaders (VPs, directors, managers) interested in understanding the essentials of A/B testing to accelerate innovation

Program managers focused on growth, revenue, conversions, and prioritization who will understand how to better design robust clear metrics

Data science managers and scientists who do not need or want the 2nd part of https://bit.ly/ABClassRKFlagship (or prefer videos)

Organization leaders (VPs, directors, managers) interested in understanding the essentials of A/B testing to accelerate innovation

Program managers focused on growth, revenue, conversions, and prioritization who will understand how to better design robust clear metrics

Data science managers and scientists who do not need or want the 2nd part of https://bit.ly/ABClassRKFlagship (or prefer videos)

Anyone interested in making the organizations more data-driven and efficient with fewer severe incidents

What you will learn from this course

Understand and internalize the humbling reality that we are poor at assessing the values of ideas: most ideas fail!

You will hear multiple real memorable stories and examples, many that don't make it to books or articles. These were chosen from over 20 years of experimentation. You'll have the data to show that the poor success rate is documented across multiple organizations; expected it!

Understand the key advantages and limitations of A/B testing

Understand key concepts like causality, hierarchy of evidence, and key organizational tenets required for effective experimentation.

Learn how to design metrics and the Overall Evaluation Criterion

Designing metrics is hard. There is a hierarchy of metrics and perverse incentives. The most important metrics comprise of the OEC - The Overall Evaluation Criterion. We will look at good and bad examples.

Learn how to designing trustworthy A/B tests

Getting numbers is easy; getting numbers you can trust is hard. You'll understand common pitfalls and how to design reliable and trustworthy tests.

Learn how to generate and prioritize ideas

You'll understand the key tradeoff between the cost and fidelity of different methods. You'll learn about the key concept of EVI: the Expected Value of Information. You'll learn about different prioritization frameworks.

Course syllabus

4 live sessions • 8 modules • 19 lessons

Expand all modules

Week 1
Week 1
Nov 6—Nov 12
Modules
Introduction and Motivation
Introduction and Motivation
3 Lessons
(44 minutes)
(44 minutes)
Tenets, advantages & issues
Tenets, advantages & issues
3 Lessons
(55 minutes)
(55 minutes)
Metrics and the OEC
Metrics and the OEC
4 Lessons
(71 minutes)
(71 minutes)
Week 2
Week 2
Nov 13—Nov 17
Events
Nov
13
Session 1 - Option 1
Session 1 - Option 1
Mon, Nov 13, 4:00 PM - 5:00 PM UTC
Nov
14
Session 1 - Option 2
Session 1 - Option 2
Tue, Nov 14, 1:00 AM - 2:00 AM UTC
Nov
16
Session 2 - Option 1
Session 2 - Option 1
Thu, Nov 16, 4:00 PM - 5:00 PM UTC
Nov
17
Session 2 - Option 2
Session 2 - Option 2
Fri, Nov 17, 1:00 AM - 2:00 AM UTC
Modules
Monday Live Session
Monday Live Session
1 Lesson
Designing and Interpreting Experiments
Designing and Interpreting Experiments
3 Lessons
(47 minutes)
(47 minutes)
Twyman's Law, Trust, Idea Generation and Prioritization, and Ethics
Twyman's Law, Trust, Idea Generation and Prioritization, and Ethics
2 Lessons
(40 minutes)
(40 minutes)
Additional Topics
Additional Topics
2 Lessons
(26 minutes)
(26 minutes)
Thursday Live Session
Thursday Live Session
1 Lesson

100% Customer Satisfaction Guarantee

Not satisfied after seeing some videos? Get a full refund within 48 hours of the course start

4.8

(3 ratings)

Attendee Testimonials: What People Said (from the longer course)

Ronny made the content amazingly easy to consume...I can't recommend it enough

Dylan Lewis

Experimentation Leader at Atlassian

If you're interested in running A/B tests and learning the science behind them, my highest recommendation is to take Ronny Kohavi's cohort-based course on the topic. (The book Ronny co-authored is tied for first on my reading list too.)

Ryan Lucht

Director of Growth Strategy, Cro Metrics

The entire course and the way it was delivered was amazing - had tons of learning, especially on where we could go wrong. Examples followed by key concepts was great. Really appreciate Ronny taking time to answer each question even after the session.

Pavan Gangisetty

Staff Data Analyst @ Intuit

The culture of Q&A during the session. The depth and width of the experimentation topic. Really eye-opening learnings from Ronny. Thank you, thank you, thank you!

Han Dong

Sr. Business Intelligence Engineer @ Credit Karma

I love Ronny's stories. I think that apart from the learning, two of the most valuable things from this class are: 1) Stories because you will remember them and they will come in handy and 2) Links so that you can dive deeper.

Ishan Goel

Lead Data Scientist @ Wingify

50% of what I've learned myself about experimentation is wrong. The other half I learned from Ronny.

Deborah O'Malley

Founder & CEO @ GuessTheTest, named a "top-10" digital marketing experts to follow

Ronny's tons of experience is valuable and hearing it firsthand is amazing.

Jialin Huang

Director, Customer Knowledge & Strategic Insights @ Fidelity

Favorite part: Practical info and learnings that weren't in the book but were probably not easily publishable.

Scott Rome

Senior Principal Engineer @ Comcast

This is an amazing opportunity. I think of all of the times I have questioned, “does X make sense?” I saved up all of those questions. I’ll tell you that Ronny Kohavi covered MANY of them in his lecture content (so I didn’t have to ask those). It was so powerful to just ask directly. Easily saved me hundreds if not thousands of hours of research.

Aaron Gasperi

Principal product manager, Walmart

Just had an incredible experience as the entire leadership team at Vinted Marketplace joined me in the 'Accelerating Innovation with AB Testing' course by Ronny Kohavi himself! 💪🔥 …Let's take our experimentation game to the next level!

Manuel de Francisco Vera

Sr. Director, Data Science & Analytics, Vinted

Companies with two or more people that took my courses

Top: Companies sorted by approximate market value. A/B Vendors: sorted by alphabetical order

Meet your instructor

Dr. Ronny Kohavi

Ronny Kohavi was an executive at Amazon, Microsoft, and Airbnb and has over 20 years of experience running A/B tests and leading experimentation teams. He loves to teach, and his papers have over 60,000 citations. He co-authored the best-selling book: Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing (with Diane Tang and Ya Xu), which is a top-10 data mining book on Amazon. He is the most viewed writer on Quora's A/B testing and received the Individual Lifetime Achievement Award for Experimentation Culture in Sept 2020.

Ronny holds a PhD in Machine Learning from Stanford University.

See more at http://www.kohavi.com

Course schedule

Over 4.5 hours of videos + 2 hours live

Week 1: Watch 10 videos of ~20 minutes each (total 2:50 hours)
Watch videos on your own
- Ask questions on community forum channel
Week 2: two one-hour live sessions, 7 videos (total of 1:53 hours of video)
Attend live sessions
- Live session Monday
- Watch 1:53 hours of videos on Tuesday/Wednesday
- Live session Thursday

Free resource

A/B testing book Chapter 1

Interested in reading chapter 1 of my book: Trustworthy Online Controlled Experiments : A Practical Guide to A/B Testing?

Get this free resource

Be the first to know about upcoming cohorts