Practical AB Testing for Product Managers
Andrew Mende
Sr ML Product Manager Booking.com, MBA
What the PM needs to know to run experiments
To run A/B experiments on your product, you donāt need deep knowledge of statistics. You are a PM ā you operate on a different level. You need to understand the core principles of testing.
ā Why learn with me?
Iāve been running A/B experiments as a product manager for 10 years. Iāve designed, launched, and took decisions on more than 1,000 experiments across different products with millions of visitors such as booking.com.
I will explain exactly what the role of a PM is in a data-driven team. PMs donāt use formulas, but they need to understand the concept of statistical power ā and, above all, how to test what matters most.
ā What makes this course different? Practical learning!
You donāt learn by watching videos or reading slides. You learn by solving real-world tasks and getting feedback on your decisions.
Thatās why my course is built around practice and solving cases. We are going to design experiments and make decisions.
ā A/B experimentation is a skill you must have
Some product decisions canāt be made from intuition, empathy, or product vision alone. They need to be based on data. If you want to work on a big, mature product, experimentation must be part of your core toolkit.
What youāll learn
Apply experiments where they are really needed
What makes a good hypothesis
How do we make decisions based on data
Pros and cons of Š/B testing a product feature
How long should I run my experiment
Which metric to target when testing a product improvement
How do PMs use the sample size calculator in practice
What are most common mistakes product teams make when taking decisions
Reading beyond the primary metric, how to evaluate the hypothesis
Successful experiment? Congrats. What are the next steps to leverage learnings?
How to manage several experimentation lines
Balancing between exploration and exploitation to maximize outcomes
Design an A/B experiment for a real-life usecase
Practice reading metrics in a complex situation
Take a decision and back it up with a short rationale
Who this course is for
Product Managers
This course is written by a PM for PMs, I assume basic knowledge of key product metrics such as conversion, retention etc.
PMs who want to work on large scale products
What to develop your career in big tech? This is a skill employers are looking for.
Product Managers with science-based approach
Not event the Hippo can argue with learnings backed by data and experimentation
Prerequisites
Basic Product Management skills
I will assume that you have some experience leading a product team and you are comfortable with managing backlog and the roadmap.
What's included
Live sessions
Learn directly from Andrew Mende in a real-time, interactive format.
150+ exercises and 3 hands-on projects on A/B experimentation
You will practice designing experiments, doing pre-experiment power calculation, analyzing results and taking decisions in a simulated environment.
Lifetime access
Go back to course content and recordings whenever you need to.
Community of peers
Stay accountable and share insights with like-minded professionals.
Certificate of completion
Share your new skills with your employer or on LinkedIn.
Maven Guarantee
This course is backed by the Maven Guarantee. Students are eligible for a full refund up until the halfway point of the course.
Course syllabus
Week 1
Apr
14
Workshop #1 Working with hypotheses, not features
Apr
16
Workshop #2 Understanding statistical power
Week 2
Apr
21
Workshop #3 Practice taking data-driven decisions
Apr
23
Workshop #4 Managing portfolio of experiments
Free resources
Schedule
Live sessions
2 hrs / week
Two hands-on workshops per week, practice in a group with like-minded PMs.
Tue, Apr 14
3:30 PMā5:00 PM (UTC)
Thu, Apr 16
3:30 PMā5:00 PM (UTC)
Tue, Apr 21
3:30 PMā5:00 PM (UTC)
Thu, Apr 23
3:30 PMā5:00 PM (UTC)
Projects
1-3 hrs / week
Guided hands-on projects in simulated environment, test your skills in a safe space.
Testimonials
The course gave me the knowledge and skills to effectively design and analyze AB tests. I am very glad that I can immediately apply the acquired knowledge in my work. During the course, we stopped the crookedly launched AV tests, choose the metrics, counted the sample sizes, and launched them with a clear conscience and in accordance with all the rules :)
Helena
Data AnalystThe course provided me with a fundamental understanding of running experiments on digital products and highlighted possible errors and mistakes we could have fronted otherwise. This course covers topics like: when to use A/B experiments, confidence intervals, statistical test and power, making a decision on an A/B test, and many more.
Djalil
Product ManagerThe course helped me get rid of the inferiority complex š¤£. It showed that everything is done through a product hypothesis despite the strict mathematical apparatus. At the forefront, we have not the testing process, but the product and its metrics. I got a specific track with the instrumentation that I use.
Piter
Product Manager