JAN-2025

Weak A/B test design can hide great ideas beneath insignificant results. Learn 10 checks to avoid throwing away valuable results.

Jakub Linowski

Change & Measure: 10 Checks For Designing Better AB Tests

Microsoft

 Designlab

Reverb

LinkedIn

Viva Real | São Paulo SP

10 test design checks to improve the quality of your experiments: ❌ Loosely set triggers dilute your test, extending its duration and causing a loss of power.✔️ Learn to set better test triggers. ❌ Underpowered tests are likely to lead to insignificant results.✔️ Learn to estimate and set realistic Minimum Detectable Effects using historical data. ❌ Compromise and confounding often washes out effects.✔️ Learn to separate and structure tests with minimum confounding. ❌ Lack of iteration and giving up on an a/b test too quickly leads to wasted effort.✔️ Learn to design your a/b tests in ways that enables you to iterate. ❌ Lack of supporting metrics limits your ability to make correct decisions.✔️ Learn to include custom metrics in your a/b tests that reinforce your hypothesis. ❌ Lack of segmenting may hide the effect in the general sample.✔️ Learn to setup your tests with custom segments that may show effects in smaller subsets of users. ❌ Some changes may be too weak for you to see any difference.✔️ Learn to setup your tests with different intensities. ❌ Some ideas may be better than others.✔️ Learn to explore and select ideas from historical data to increase your odds of success. ❌ The effect may be too small from a single change.✔️ Learn to design compound experiments that combine multiple changes with positive probabilities. ❌ Not all experiments need to test for superiority which demands more power.✔️ Learn to design non-inferiority tests in some situations.

Design better experiments with fewer insignificant & throwaway results

You're a UI, UX or CRO Test Designer who is frustrated with too many insignificant experiment results.

You're an Experimentation &amp; Optimization Strategist responsible for leading A/B testing programs your team and your clients can trust.

You're a CRO, Growth or Ecommerce Manager responsible for increasing metrics such as revenue and sales with impactful A/B test results.

Who this course is for

By the end of this 1 week course, you'll:

Learn to check your A/B test designs to minimize the odds of insignificant and false positive results

Design A/B tests with more sensitivity

Have fewer throw-away test results

Speed up your experiments

Receive a downloadable template with the A/B test checklist you can use on your projects

Learn from 10-20 common A/B test design scenarios and setups

Join an optional A/B test review session where we'll discuss actively designed experiments and apply the learnings

Provide a brief description for each module as a way to sell the value of what students will be learning throughout the course.

First Module

Second Module

Third Module

Course syllabus

Vice President and Technical Fellow | Data Science, Engineering | Controlled Experiments | Ex-Airbnb, Ex-Microsoft, Ex-Amazon

Author of "Statistical Methods in Online A/B Testing" | Founder of Analytics-Toolkit.com

Sr. Director Personalization & Experimentation at Signet Jewelers

What people are saying

Meet your instructor

Tuesdays & Thursdays

Friday: A/B Test Reviews

Course schedule

Learn from real A/B test design scenarios

This course builds on live workshops and hands-on projects

Active hands-on learning

You’ll be interacting with other learners through breakout rooms and project teams

Interactive and project-based

Join a community of like-minded people who want to learn and grow alongside you

Course completed by

About this course

Get notified about the next cohort

Learn from live sessions