22 Sept 2025 Advanced

Experimentation × UI Design

Jakub Linowski

Learn advanced A/B-testing topics from a world-leading expert. Ten topics in three days of 2 hours each.

⚠️ Warning: Advanced course 

Author of best-selling A/B testing book

Microsoft

Airbnb

Amazon

Dr. Ronny Kohavi

Technical fellow and VP at Microsoft & Airbnb, co-author of A/B testing book

Advanced Topics in Practical A/B Testing

Kohavi

<h3>Why I enrolled</h3>
Discover and or refine experiment decision heuristics / protocols. Ex:<ul class="list-disc ml-8"><li>can we have a hybrid test design: fixed horizon + sequential?</li><li>for basic UI / copy experiments (with low implementation / maintenance cost), should we default to raising the p-value / or running non-inferiority tests?</li></ul>Questions:<ul class="list-disc ml-8"><li>Reading your "Pitfalls of Long-Term Online Controlled Experiments" paper, when we are experimenting with stable USERIDs (example: in app) I understand that the issue of sample cross-contamination / cookie clobbering disappears?</li><li>For personalization class experiments - I have an emerging idea of an A/B/P setup which I was curious if anyone else ran in the past (A = low end value of a treatment variable; B = high end value; P = personalized treatment; end goal to have P outperform both A and B). If time permits. </li></ul>



<h3>What I’ll apply</h3>
<ul class="list-disc ml-8"><li>CUPED only for repeat visitors</li><li>More power calculations</li><li>Being careful of causes of Simpson's Paradox</li><li>Hybrid test designs (fixed horizon / sequential)</li></ul>

Certificate

Learn more about this course