Maven

How to Create a Culture of Experimentation

Lyft

Patreon

Imperfect Foods

Mozilla

Wyzant

Adam Fishman

Growth Advisor & Interim SVP of New Products at Mozilla (previously Patreon and Lyft)

Gagan Biyani

Co-founder of Maven

Strategies to set measurable goals, provide structure with flexibility, and let your employees figure out the rest

How to lead with goals & autonomy

Examples of effective brainstorming sessions from companies like Patreon and Lyft

How to run open brainstorms, with constraints

Ways to achieve a culture of experimentation by rewarding failure and making it okay to "take the L"

How to reward failure and make it less risky

Please read Adam's article before the lesson: https://bit.ly/3RO2UsD

📚 Recommend reading

Through multiple real examples of well-run experiments and real stories at Microsoft, Amazon, and Airbnb, you will see the humbling reality that we are terrible at assessing the values of ideas. Trivial changes can be surprisingly useful, whereas large efforts often fail. Accelerate innovation by building Minimum Viable Products and Features (MVPs) and make the organization evidence-based and humbler, as it adopts and learns to use evidence from the gold standard in science: the controlled experiment. You will understand the challenges in designing and running trustworthy controlled experiments, or A/B tests, including the importance of the Overall Evaluation Criterion (OEC), scaling, pitfalls, and Twyman's law. The 2nd week covers additional topics, some more technical, including cultural challenges, institutional memory, maturity model, observational causal studies, offline evaluations, AI/Machine learning and triggering, scaling, build vs. buy, challenges, and requested topics. The advanced topics course is a follow-on to this: https://bit.ly/AdvancedABFS

Empower your organization to be data-driven and innovative

<ul><li>Introduction to controlled experiments, or A/B testing</li><li>Interesting experiments – you’re the decision maker</li><li>Organizational tenets</li><li>Advantages and limitations of controlled experiments</li></ul>

Introduction

<ul><li>Interesting example</li><li>End-to-end experiment: Running and Analyzing an Experiment</li><li>Metrics</li><li>The OEC</li></ul>

End-to-end example, and Metrics and the OEC

<ul><li>Interesting example</li><li>P-values and statistical power</li><li>End-to-end experiment: speed matters</li><li>Twyman’s Law and Trustworthy Experimentation</li><li>Generating ideas, prioritization, ethics</li></ul>

P-values and power, End-to-end Example 2, Twyman's law and Trustworthy Experimentation

<ul><li>Interesting example</li><li>Cultural challenge, Institutional memory, maturity model</li><li>Observational causal studies and quasi-experimental designs</li><li>Pitfalls in observational studies</li></ul>

Cultural challenges, Prioritization, AI/Machine Learning, Complementary Techniques and Observational Causal Studies

<ul><li>AI/Machine Learning model, and triggering</li><li>Leakage and interference</li><li>Bayesian A/B Testing</li><li>Experimentation Platform and scaling</li><li>Build vs. Buy</li><li>Challenges</li><li>Requested Topics</li></ul>

Advanced topics, challenges, and requested topics

Course syllabus

Three x 2-hour sessions in week 1 on Monday, Tuesday, and Thursday

Week 1: Mon, Tue, Thu

Two x 2-hour sessions in week 2 on Monday and Thursday

Week 2: Mon, Thu

Optional Q&A

Course schedule

Data science managers and scientists will be able to design and interpret the experiment results in a trustworthy manner

Program managers focused on growth, revenue, conversions, and prioritization will understand how to provide the org with robust clear metric

Engineering leaders will be able to make the organizations more data-driven and efficient with fewer severe incidents through A/B tests

Who is this class for?

What you will learn from this 10-hour course

You will hear multiple real memorable stories and examples, many that don't make it to books or articles. These were chosen from over 20 years of experimentation. You'll have the data to show that the poor success rate is documented across multiple organizations; expected it!

Understand and internalize the humbling reality that we are poor at assessing the values of ideas: most ideas fail!

Understand key concepts like causality, hierarchy of evidence, and key organizational tenets required for effective experimentation.

Understand the key advantages and limitations of A/B testing

Designing metrics is hard. There is a hierarchy of metrics and perverse incentives. The most important metrics comprise of the OEC - The Overall Evaluation Criterion. We will look at good and bad examples.

Learn how to design metrics and the Overall Evaluation Criterion

Getting numbers is easy; getting numbers you can trust is hard. You'll understand common pitfalls and how to design reliable and trustworthy tests.

Learn how to designing trustworthy A/B tests

Learn about the cultural challenges, the humbling results (most ideas fail, pivoting, iterating, learning), institutional memory, ideation, prioritization, experimentation platforms

Learn about the cultural challenges

When building AI or machine learning models, using A/B testing and triggering to evaluate the models that were built offline based on historical data

Learn about the relationship to AI and Machine Learning Modeling

When you can't run an A/B test, quasi-experimentation methods and the risks of observational causal studies

Learn about complementary quasi-experimental techniques

What are key challenges and open questions in the field

Learn about existing challenges

If there is something specific you want to cover, there is time allocated for topics voted by the audience to discuss

Learn about YOUR topic of interest

The course focuses on developing the intuition and common misunderstandings, without the details of the statistics, which you can find in many books. We cover p-values, statistical power, and triggering.You can go as technical as you want in the Q&amp;A and community discussions

Technical to your needs

Learn from a world-leading expert how to design and analyze trustworthy A/B tests to evaluate ideas, integrate AI/ML, and grow your business

Dr. Ronny Kohavi

2 Dec 2024

3 March 2025

Do you ever feel stuck, like your life is on autopilot? That’s exactly where I was. I was an introverted software engineer, overweight, inactive, and spending most of my time on the couch. My well-paying job became my identity, but deep down, I felt lost and directionless. Then everything changed. One day, inspired by the story of adventurer John Goddard, I decided to set real, meaningful goals—and it transformed my life. I lost 50 pounds, became a confident public speaker, ran a marathon in under four hours, moved to the U.S., joined Meta, and even published a book. Goals didn’t just change my habits—they redefined who I was. I wish I’d had a roadmap back then, a simple course that would have shown me how to set and achieve goals correctly. That’s why I created this course—for you. This step-by-step course is designed to give you a practical and powerful framework for turning your personal and professional dreams into reality. Together, we’ll follow a simple, 7-step process to help you:• Define your goals with clarity.• Build a plan that keeps you motivated.• Overcome obstacles and setbacks.• Transform your vision into action. Whether you want to advance in your career, improve relationships, or accomplish a lifelong dream, this course will give you the mindset and tools to take control of your future. It’s time to stop dreaming and start achieving. Join me on this journey and discover how setting goals can unlock a life full of purpose, success, and happiness. Start today. Your best self is waiting! COURSE CONTENT:• How to identify meaningful goals• How to increase your odds with accountability• How to smartly define milestones• How to craft masterpiece days with intention and attention• How to bounce back after setbacks• How to quit strategically• How to enjoy the journey THE COURSE ALSO INCLUDES:• Review and self-reflection• Real-life examples• Hands-on exercises to identify and plan your goals• Frameworks to help you execute your plan• Plus: Homework to put concepts into practice (optional)• All 7 Zoom workshops are live, taught by Pramoda All the sessions will be recorded so you can refer back to the material. WHO WILL BENEFIT MOST FROM THIS COURSE:• You are ambitious, and you want to sharpen your ability to achieve your goals• You are accountable and willing to share and learn from your peers• You are self-aware, open-minded, and intellectually honest WHO THIS IS NOT FOR:• This is not for you if you are looking for 1:1 coaching, but I will do plenty of Q&amp;A and ask for volunteers to submit your situations/examples to discuss as a group• This is not for you if you believe there is nothing more you can learn about achieving your goals• This is not for you if you want a quick fix or shortcuts for your goals This is a 7-week incremental course designed for an immersive experience.&nbsp; You will develop new habits and approaches to achieving your goals.&nbsp; In these seven weeks, I aim to improve your competence and confidence and make you a fully independent goal achiever.

Goals shift "You" from FOMO(Fear Of Missing Out) to JOMO(Joy Of Missing Out).

 

Provide a brief description for each module as a way to sell the value of what students will be learning throughout the course.

First Module

Second Module

Third Module

Saturday live sessions

You are looking for growth and expansion in your life and career.

You are keen to achieve meaning and happiness through personal and professional goals.

You want to shift from “Fear of missing out” to “Joy of missing out.”

This course is for you if:

What you’ll get out of this course

so you can easily accomplish your personal and professional goals, which means you can be more fulfilled.

Learn a practical and repeatable framework for goal achievement 

so you gain crystal-clear clarity on your goals, which means you can stop comparing yourself to others.

Identify your short-term(1-year) and long-term(3-5 years) goals 

so you can achieve micro goals with less effort and more enjoyment, which means your journey can be smooth and enjoyable instead of painful.

Master the art of defining milestones and minifying them

so you can achieve more with less effort which means you no longer have to rely on sheer willpower.

Create a personal plan and daily habits to unleash the best version of “You”
 

so you no longer feel like a lone warrior, which means you have a supportive community to rely on for assistance and encouragement.

Collaborate with other growth-oriented people who can hold you accountable
 

Life improves when you take control—set goals and achieve everything you’ve ever dreamed of.

Pramoda Vyasarao

Author and Leadership coach

Cohort 1

Cohort 2

COURSE CONTENT:8 Hours over one weekend, and 2 bonus hours before and after, combining presentations and case studies, interactive exercises in small groups and self reflection time. OVERVIEW: You are already comfortable managing others, but to get to the next level of leading large teams, you are aware that you need to be able to identify and drive transformative opportunities for the business, assess and lean into the risks associated with them, overcome your own fears of putting your ideas on the line, and inspire followership from others. You also know that this level of leadership can be exhausting - and require resilience in the face of push back or results taking longer to achieve than you hoped. Its tempting to "put off" the time it takes to work on these topics, but if you can invest the time now, it will make your leadership impact, visibility and effectiveness significantly stronger, by not having your own fears, or perceived weaknesses getting in your way. KEY TOPICS INCLUDED: 1. Establish your leadership advantage.- How to stop comparing yourself to others and instead lean into your own unique strengths as a leader- How to overcome imposter syndrome and tap into your courage to be confident leading at a higher level- How to get comfortable admitting what you don't know - and inviting others in to help as your support system to grow your confidence- How to get comfortable leading others on a larger team who's functional skills are outside your own knowledge and/or who have more age/experience than you- How to inspire and influence others to come along with ideas you are championing 2. See the opportunities that others won't. - Finding the gap - how to leverage foresight instead of hindsight in the pursuit of breakthrough strategies- How and when to tap into your instincts, when data isn't available for decision making- How to understand when a breakthrough opportunity is achievable with your team and your resources 3. Overcome your fears.- How to understand what holds you back from making bold moves- How to understand your tolerance for risk in relation to others to get confident to make more transformative moves- How to bounce back and recalibrate when things do not go as you expect 4. Navigating the politics. - How to get the confidence to invite dissent and assess the input and feedback from others who do not support your bold moves- How to go from trying to "convince" others, to "co-creating the future" with others.- How to read the room and ensure peer and team discussion is transparent and constructive- How to react when you know colleagues are not supporting your plans- How to make decisions when your team are disagreeing with each other and need you to break the tie 5. Fuel your endurance. - How to understand your unique energy needs to ensure that you prevent burnout during times of great stress- How to understand the different energy needs of others on your team so that you can work together to meet the collective team's needs 6. Build a culture of high performance fun.- How to leverage your own unique style to inspire others and build authentic followership- How to set your team "house rules" to establish a culture of high expectations and high results- How to build an inclusive team that is comfortable with dissent This is a course that is designed to get you out of your comfort zone. To get to the senior levels, it is not enough to deliver on your own existing KPIs, it requires you signaling that you are the kind of leader that is proactive and has courage, conviction and the ability to influence. This course will get you out of the text book and in to real life examples to "road test" how you would react. Course material and advice is based on real life case studies from Sarah's business experience moving from a manager of a small team of 5 individual contributors at one company, and then moving her way up to leading the complete transformation and turnaround of the $5Bn global Gatorade business less than 10 years later. Sarah is a proven coach and developer of talent, having managed teams as small as 5 and as large as 15,000. She led the reinvention and turnaround of the $5Bn Gatorade business as its global president, before leading the digital transformation of Equinox Fitness Clubs as president. Sarah has also taught workshops to over 500 young professionals, with an average NPS rating in the 60s. TESTIMONIALS FROM STUDENTS WHO HAVE TAKEN SARAH'S IN PERSON WORKSHOPS "It was a really fun two days taking the time to reflect on really important "stuff" that I normally don't make time for. It's so easy to get wrapped up in the day to day grind of work without pausing to take that time to check in with yourself. I'm also rarely surrounded by such an inspirational group of people who have relevant complementary skillsets and experiences to my own. I find that hearing others' goals pushed me to think bigger and take on even more. Thank you!Head of Client Services for an Innovation Agency "Loved this! Will definitely recommend to friends. Loved the interactive exercises - especially the debate one and the compliment one. The discussion about my blind spots (figure out weaknesses and the anti-you) was SO helpful. The group made me think about things I never would have realized. Exercise about pain training was so helpful and almost therapeutic to hear others fears and commiserate."Product Leader "I loved it. Great format with lots of variety. I came away feeling motivated to make change, supported by those around me, and inspired by everyone in the room. I would be most likely to recommend this to someone who has been considering a leap, or who has tangible goals in mind."Director of Strategy "This was a great experience. It flew by, which doesn’t happen with these types of workshops. I really liked the layout of content and the perfect combination of case studies and larger group talk. You created a safe space for everyone to be vulnerable, supportive, encouraging and excited."Director Programming and Operations "Very challenging, ahead of the game and awe-inspiring workshop."IT Strategy Consultant

An interactive weekend immersive, that  uses real situations to teach students.


Pre course meet n greet session for students

Friday December 6th


<ul><li>All sessions are recorded</li><li>Students have lifetime access</li></ul>

Saturday December 7th and Sunday December 8th


Post course office hours for students

Friday December 20th


You have 8-10 years of experience and aspire to, or have landed your first leadership role with a large team and want to assert your vision

You are shifting from managing individual contributors to leading through managers and you want to create a culture of high performance fun

You need to step onto a bigger leadership stage and want to know how to establish your vision to make transformative high impact moves

Who is this course for

By reviewing insights from personality inventories and peers, you'll establish a clear framework that you can turn back to that articulates your most confident leadership style so that you can cultivate courage to lead others with skills and experiences beyond your own.

Confidence and strength to lead. Identify your competitive advantage as a leader to leverage your strengths and support your weaknesses.


Through an interactive exercise, you will explore the differences in your unique style for thinking and sharing your ideas, so you can develop your plan to create a culture of high performance with others, to help you drive high impact results.

Insights on building a high performing team. Understand how your style best plays, to improve your ability to get results through others.


Through an interactive case study based game, you will learn how to hone your instincts, and when to use them to see opportunities that others can't so that you are able to break through and add value to the business that others may not.

Strategies for identifying opportunities for growth. How to find the gap - and see opportunities for yourself, your team and your company.


Through a framework for group discussion, you will develop an understanding of your tolerance for risk in relation to others, and you will develop a personal strategy for overcoming your fears of making mistakes, which is critical when leading at the senior level.

Methods for facing your fears. Analyze what holds you back and establish strategies for risk taking, that enable you to drive change.


Through real life case studies you will learn how to navigate politics, influence others, turn to team mates for support when your fears kick in, and develop a willingness to admit that you don't have all the answers - a requirement for creating followership of large teams.

Techniques for top down and bottom up communication. How to get buy in, navigate politics and influence stakeholders to support your vision.


You will participate in an interactive case study assignment to teach you how to assess the long range viability of your greatest transformational moves, as it is critical at the senior level to understand how to evolve and sustain your biggest ideas.

Clear understanding of social cues. Apply a framework to know when to play harder, to pivot or to walk away from ideas you have championed.


You will spend time understanding your own needs around health, wellbeing and energy management so that you can prevent burnout. You will develop a framework to create a set of commitments to yourself that ensure you're endurance for the long leadership road ahead.

Strategies for sustaining yourself. How to establish the unique rituals that you will need to prioritize your own health and wellbeing.


Learn to lead boldly, assess risks, and drive change. Gain confidence to inspire large teams and seize breakthrough leadership opportunities


Sarah Robb O'Hagan

We see in spades what was once thought to be science fiction. Flying. The internet. Artificial intelligence. If you dropped someone from 100 years ago into today, they would think much of our existence to be&nbsp;magic. Only the wildest of imaginations could have ever come up with inventions so foolishly brilliant. When you realize that everything around us was once just a thought — often coupled with a sprinkle of audacity — there’s no going back. “Life can be broader once you discover one simple fact: everything around you that you call life was made up by people who were no smarter than you. And you can change it… you can influence it… you can build your own things that other people can use. Once you learn that, you’ll never be the same again.” — Steve Jobs Throughout history, we’ve seen mavericks emerge—Da Vinci, Einstein, Curie—they left clues, principles, and perspectives that we can harness in our own lives, even in the everyday. Bold ideas and brave actions aren’t reserved for history books, and the power to reshape the world isn’t reserved for a select few. Each of us has the capacity to carve out our own path and leave a mark. The question is — do we believe it’s possible for us? A prime example is the four-minute mile. Once considered beyond human capability and thought to pose severe physical risks, the goal created a psychological barrier for runners for over 70 years. Once broken, over a thousand runners achieved the same feat. Fascinating, isn’t it? What’s something so special about your life today, that a past version of you could never have imagined? Well, here’s a story of mine… I used to struggle with sharing myself with the world — and public speaking was the most terrifying thing in the world to me. Shaking and anxious, I brute forced my repetitions into excellence. I published a book and keynoted at Harvard. And honestly, I thought it was an accident. Some kind of fluke. Until I was invited to speak on a panel at the World Economic Forum. I have the flexibility to live abroad seasonally. I’ve worked with over 25 ventures. As of today, I run 3 businesses. I work out most days and usually get about 7-9 hours of sleep. My life has been a whirlwind string of impossibilities. No amount of strategizing could have gotten me here — a life I hardly recognize as my own. It all comes down to mindset. We have the power not just to shatter our own “limitations” but to&nbsp;entirely transform the sense of what’s achievable&nbsp;— much like many of the humans we'll learn about in this course. This course is for you if you’ve been dancing around the perimeter of the person you’ve always wanted to be. We'll take away with our own personal philosophies of impact — exploring ideas all the way from an anthropocentric view of genius, achieving flow states, to dopamine detoxing. And more importantly, we’ll walk away having remembered that the power to change our own lives has always been within us. Given that our work has the potential to outlive us by generations, I sincerely hope to welcome you to this transformative experience. And if you do well, you can do good. In the near future, we will experience one of the&nbsp;greatest wealth-creation events in human history. The rise of new industries and technologies will create immense abundance for those who are involved in creating, developing, and investing in them. Our decisions have far-reaching consequences, influencing the course of industries and societies. We must embrace this power with a sense of&nbsp;responsibility&nbsp;and&nbsp;compassion. “The greater danger for most of us lies not in setting our aim too high and falling short; but in setting our aim too low, and achieving our mark” - Michelangelo That leaves us with:&nbsp;Who will you be and what will you create?

In observing the world around us…

A bespoke 15-minute session to help you move through your blocks and limiting beliefs to unlock your most successful self yet Limited spots available; first come, first serve.

🔥BONUS #1: Hop On A 1:1 Session With Arooj

Elevate your journey of entrepreneurial self-discovery with a digital journal featuring exclusive tools, profound insights, and invigorating exercises, crafted to enhance your well-being.&nbsp;

🔥BONUS #2: Digital Journal

Reference-able recaps for all sessions

🔥BONUS #3: Key Takeaways

And it's my personal favorite, by a mile.

🔥BONUS #4: This one... this one's a surprise!

🔥🔥BONUSES 🔥🔥

This course takes place over the weekend to accommodate busy work schedules. All sessions will be recorded! Across Time Zones9:30 AM - 12:30 PM San Francisco 12:30 PM - 3:30 PM New York5:30 PM - 8:30 PM London8:30 PM - 11:30 PM Dubai

An Immersive, One-Day Virtual Workshop

<ul><li>Engaging group instruction with ample lively discussion</li><li>Intellectually stimulating conversations with peers</li><li>Guided workbook with practical resources to weave into your mindset rituals</li><li>Optional readings to deepen your experience and understanding</li></ul>

A Symposium For The Intellectually Curious

Course Schedule

Aspiring polymaths interested in flow state and intrigued by the hidden reality of geniuses

The intellectually curious investor, entrepreneur, or leader crafting a legacy of impact

World-changing leaders interested in cultivating transformative self-awareness

What you get out of this course

<ul><li>Develop a Planetary &amp; Anthropocentric View of Genius</li><li>8+ Genius Case Studies</li><li>Nature vs. Nurture: Society's Role in Cultivating Genius</li><li>Leveraging Creativity, Failure, and Flow States</li><li>Master 12+ Powerful Mental Models That Fuel Genius</li><li>Construct An Elastic Understanding of Reality</li></ul>

PILLAR 1) The Secret Nature of Genius

Cultivate a personalized “genius framework” that enables you to solve complex problems and innovate meaningfully

<ul><li>Reconstructing Your Self Concept (Impact Identity)</li><li>Designing For Nurture: Nudging Your Environment To Allow For Genius</li><li>Evaluating Philosophy, Genius, and Ethics</li><li>The Power of Lived Experience: Exploring Contrasting Perspectives</li><li>Designing a Lasting Legacy for Impact</li></ul>

PILLAR 2) Defining Your Personal Philosophy of Impact

Achieve clarity and scale your influence by crafting a personal framework for powerful results

<ul><li>Uncovering The Neuroscience of a Winning Mindset</li><li>Dopamine Detoxing</li><li>Visualization for Transformation</li><li>Journaling for Growth</li><li>Create Your Meditation Toolkit</li><li>Designing Your Breath Work Flow</li><li>Abundance Principles Rooted in High Performance Psychology</li><li>Linear vs Exponential Growth Systems</li></ul>

Lightning Lessons

How to Create a Culture of Experimentation