Alina Bezchotnikova k

Why Your A/B Test Results Are Probably Wrong

Alina Bezchotnikova

PhD, Principal UX designer at Aize

What is ruining your tests and how to avoid it

Main mistakes with A/B tests

Not everything can be A/B tested, let's get it right

When to A/B test

What it actually means and why should you care about it

The p-value trap

How to know that your results are trustworthy

Quick framework

Learn from a world-leading expert how to design and analyze trustworthy A/B tests to evaluate ideas, integrate AI/ML, and grow your business

Author of best-selling A/B testing book

Microsoft

Airbnb

Amazon

Dr. Ronny Kohavi

Technical fellow and VP at Microsoft & Airbnb, co-author of A/B testing book

26 Jan 2026 - noon PST

9 March 2026

Through multiple real examples of well-run experiments and stories from Microsoft, Amazon, and Airbnb, you will see the humbling reality that we are terrible at assessing the value of ideas.Trivial changes can be surprisingly useful, whereas large efforts often fail. Accelerate innovation by building Minimum Viable Products (MVPs) and features, and make the organization more evidence-based and humble as it adopts and learns to use evidence from the gold standard in science—the controlled experiment.You will understand the challenges in designing and running trustworthy randomized controlled experiments (A/B tests), including the importance of the Overall Evaluation Criterion (OEC), scaling, common pitfalls, and Twyman's Law. We cover cultural challenges, institutional memory, maturity model, observational causal studies, offline evaluation, AI/Machine learning and triggering, build-vs-buy trade-offs, challenges, and requested topics. The Advanced Topics course is designed as a technical follow-on: <a target="_blank" rel="noopener noreferrer nofollow" href="https://bit.ly/AdvancedABFS">https://bit.ly/AdvancedABFS</a>

Empower your organization to be data-driven and innovative

Internalize the humbling reality: we are poor at assessing the value of ideas

Understand the key advantages and limitations of A/B testing

Learn how to design metrics and the Overall Evaluation Criterion

Learn how to design trustworthy A/B tests

Learn about the cultural challenges

Learn about the relationship to AI and Machine Learning Modeling

Data science managers and scientists will be able to design and interpret the experimental results in a trustworthy manner

Program managers focused on growth, revenue, conversions, and prioritization will understand how to provide the org with robust clear metric

Engineering leaders will be able to make organizations more data-driven and efficient with fewer severe incidents through A/B tests

Learn advanced A/B-testing topics from a world-leading expert. Ten topics in three days of 2 hours each.

⚠️ Warning: Advanced course 

9 Feb 2026 - Advanced - noon PST

23 March 2026

We cover advanced topics not typically addressed in courses and books and leave significant time for interactions and Q&amp;A.Classes are small: typically ~15 live attendees. The course assumes you already took <a target="_blank" rel="noopener noreferrer nofollow" href="https://bit.ly/ABCourseRonKohavi">https://bit.ly/ABCourseRonKohavi</a> or have equivalent knowledge.

Improve your A/B testing capabilities and build trust

Improve your A/B Testing capabilities and trust

Discuss your own questions

Designed as a follow-on to the flagship course: https://maven.com/kohavi/abtesting. 

Take that course first, or have equivalent knowledge

Follow-on to flagship course

Good understanding of p-value misconceptions, statistical power, OEC.

Statistics of A/B Testing

Data science managers and scientists who understand the basics of A/B testing and want to better understand advanced topics

Product and program managers who want to advance their understanding of A/B testing

Acquire and develop a critical skill for product managers who are leading and contributing to AI products.

February 2026 (please delete)

Feb 2026

GenAI Product Builder @ Google, ex-Meta

Marily Nika, Ph.D AI/ML

Product Manager AI&ML @ Google

Diego Granados

Senior Data, ML and GenAI Scientist

SHI International

National Association of REALTORS

George Zoto

Senior NLP/AI Scientist

December 2025

Cohort 4

“Does it work?”... “Is it good enough?”... “Can we ship it?”...How do you answer these questions for AI products? You’re responsible for “running evals” but what does that mean?How do you choose the right metrics, interpret fuzzy results, and make a confident decision?This course gives you a framework to do just that.<ul><li>Map user value to evaluation (eval) objectives so your metrics aren’t abstract. Define success then translate it into measurable criteria.</li><li>Choose metrics you can actually maintain: capability, safety, UX friction, latency, cost and “does this reduce support tickets or increase activation.”</li><li>Set ship/no-ship thresholds you can defend to leadership.</li><li>Build lightweight workflows that work in real teams: human review where it matters, automation where it lasts, documentation that drives decisions.</li><li>Consider domain constraints (e.g., healthcare safety) and know what to avoid: silent failures, misleading proxy metrics and tests that don’t reflect production.</li><li>Tie everything to ROI: impact vs unit cost, eval coverage vs reliability, and the minimum viable monitoring you need post-launch.</li></ul>Experience AI evals through a case-based approach with a real AI product that we evaluate together.

Eliminate uncertainty in shipping AI features

Make confident ship or hold decisions for AI features

Translate user value into clear evaluation goals

Choose the right metrics for capability, safety, UX, and cost

Set defensible thresholds leadership will trust

Build lightweight evaluation workflows teams can maintain

Navigate domain constraints and avoid common failures

PMs leading AI features, growth, or platform initiatives

PMs who partner with ML teams and want to set evaluation standards

PMs who need to make clear “ship or hold” calls without doing the engineering

This is a hands-on transformation blueprint. In just 4 weeks, you'll go from "hoping it works" to "knowing it will work"

Chief Product Manager and Co-Founder

JPMorgan Chase & Co.

Market Axess

Goldman Sachs

Software AG

Anil Jaising

Anil Jaising helps product leaders thrive by unpacking innovation 

Principal Product Consultant

TiVo

Persistent Systems

Bernie Maloney

From breakdowns, through breakthroughs, to $1B and beyond

Cohort Based Course

You've watched engineering spend months building something that flopped at launch. You've sat in prioritization meetings arguing about which bet to make, with nothing but opinions and gut feel.There's a better way.This course gives you a systematic method to validate product ideas before you ask engineering for a single sprint. Using AI-powered prototyping, structured experimentation, and real market testing, you'll learn to separate winning ideas from expensive mistakes in weeks or days rather than quarters.By the end of this 4-week course, you'll have validated (or invalidated) a real product concept using techniques you can repeat on every initiative you touch.<h2>THE PROBLEM THIS COURSE SOLVES</h2>Product leaders face a credibility gap:• Competitors ship faster because they've figured out how to test before they build• Your roadmap is built on assumptions, but stakeholders expect certainty• Engineering capacity is scarce, and you can't afford to waste sprints on unvalidated ideas• AI tools exist that could accelerate validation, but you're not using them systematicallyMost PM courses teach you frameworks for thinking. This one teaches you a system for proving.

The systematic approach to validating product bets before engineering commits. 


Map and Prioritize Your Riskiest Assumptions

Prototype Without Engineering Using AI and Vibe Coding

Design Experiments That Generate Evidence, Not Opinions

Create Offers That Convert Interest to Commitment

Folks who are early career, or are meaningfully involved in content creation should not enroll

This course is NOT FOR everyone

It is almost certainly the most advanced product validation course, suited for PMs, Design, Eng, Data, etc 

This IS NOT a beginner level course

It IS NOT for folks who like to outsource their thinking to others and just follow directions or Anyone looking for theoretical frameworks 

It is FOR YOU if you like think for yourself and WANT to WIN

Product Focused Startup founders and Corporate Product Managers frustrated by the gap between their products's potential and current result

Product Directors accountable for portfolio-level outcomes, need to de-risk multiple bets and adopt validation as a standard practice

Product Managers Building new products or major pivots to prove product-market fit signals before scaling investment in high risk assumption