Steen Rasmussen

3 signs that your Google Analytics marketing data is broken.

Name

Founder Data Analytics Society, Keynote Speaker and Data Visionary.

Three things to monitor to make sure you are able to make correct marketing decision based on you GA data.

3 Google Analytics Marketing Data Quality warning signs

Google Analytics can be setup in many ways, but they are not all optimal to get the infomation you need.

Why a "by the book" setup will not tell you what you think

What can a full marketing setup secure you of insights that you can't get today.

Understand what your traffic report should tell you

Learn from a world-leading expert how to design and analyze trustworthy A/B tests to evaluate ideas, integrate AI/ML, and grow your business

Author of best-selling A/B testing book

Microsoft

Airbnb

Amazon

Dr. Ronny Kohavi

Technical fellow and VP at Microsoft & Airbnb, co-author of A/B testing book

8 Sept 2025

1 Dec 2025

Through multiple real examples of well-run experiments and stories from Microsoft, Amazon, and Airbnb, you will see the humbling reality that we are terrible at assessing the value of ideas.Trivial changes can be surprisingly useful, whereas large efforts often fail. Accelerate innovation by building Minimum Viable Products (MVPs) and features, and make the organization more evidence-based and humble as it adopts and learns to use evidence from the gold standard in science—the controlled experiment.You will understand the challenges in designing and running trustworthy randomized controlled experiments (A/B tests), including the importance of the Overall Evaluation Criterion (OEC), scaling, common pitfalls, and Twyman's Law. We cover cultural challenges, institutional memory, maturity model, observational causal studies, offline evaluation, AI/Machine learning and triggering, build-vs-buy trade-offs, challenges, and requested topics. The Advanced Topics course is designed as a technical follow-on: <a target="_blank" rel="noopener noreferrer nofollow" href="https://bit.ly/AdvancedABFS">https://bit.ly/AdvancedABFS</a>

Empower your organization to be data-driven and innovative

Internalize the humbling reality: we are poor at assessing the value of ideas

Understand the key advantages and limitations of A/B testing

Learn how to design metrics and the Overall Evaluation Criterion

Learn how to design trustworthy A/B tests

Learn about the cultural challenges

Learn about the relationship to AI and Machine Learning Modeling

Data science managers and scientists will be able to design and interpret the experimental results in a trustworthy manner

Program managers focused on growth, revenue, conversions, and prioritization will understand how to provide the org with robust clear metric

Engineering leaders will be able to make organizations more data-driven and efficient with fewer severe incidents through A/B tests

This weekend workshop is a hands-on introduction to A/B Testing designed specifically for data scientists and product managers. To truly drive impact, you must connect experiment results to business decisions, collaborate effectively with product stakeholders, and communicate findings in ways that influence roadmaps. This workshop is designed to help mid-senior data professionals build exactly those skills, sharpening your product thinking and transforming how you approach experimentation. This course offers a practical, fast-paced dive into real-world experimentation. Through interactive case-studies, group discussions, and live walkthroughs of industry tests, you will learn how to:1) Frame hypotheses that lead to actionable decisions, not just statistical tests2) Choose metrics that reflect business trade-offs — primary, secondary, and guardrails3) Interpret test results in business terms, beyond just “stat sig”4) Communicate insights clearly to PMs, leadership, and cross-functional teams By the end, you won’t just be running A/B tests — you’ll be driving better product decisions with them. We will wrap up the discussion by learning about what separates a good data scientist from a great one, and how you can continue building on your data skills.

Learn to Design and Analyze A/B Tests that Influence Product Strategy.

Course syllabus

Learn how great experiments start—with the right questions. We’ll cover hypothesis framing, choosing the right metrics, and designing experiments that align with product goals. You’ll also explore real-world examples of good (and not-so-good) test design.

Day 1: Foundations of Experimentation

We’ll dive into analyzing experiment outcomes, understanding tradeoffs, and handling ambiguous results. You’ll practice translating test results into product impact and learn how to communicate insights to influence decision-making.

Day 2: Influence Decision-Making and Drive Outcomes

Course schedule

Data scientists who want to build A/B Testing skills to share business outcomes and get interview-ready.

Growth analysts looking to design high-impact experiments and optimize key funnel metrics.

Product Managers who work closely with data teams and want to better frame, interpret and act on experiment results.

Who this course is for:

What you’ll get out of this course

Give students an idea of how they can expect to grow throughout your course. Include specificity and precise results so students can benchmark exactly what they’ll learn.

Design high-signal A/B tests by framing clear hypotheses and selecting the right success and guardrail metrics aligned with product goals.

Evaluate the impact of test results using confidence intervals, lift, and directional trade-offs to recommend feature launch decisions.

Analyze real-world case studies from Uber and Spotify to learn the applications of AB Tests to industry challenges.

Create a clear decision brief that translates experiment results into business-relevant insights for PMs, engineers, and leadership.

Learn a practical framework to evaluate real-world A/B tests, identify weak hypotheses, misaligned metrics, and missed opportunities based on product intent.

Apply a reusable A/B testing framework to critique and improve existing experiment setups based on product context and goals.

Learn to design A/B tests that drive real product decisions - from metrics and hypotheses to business impact.

Manisha Arora

Data Science Lead, Google | Founder, PrepVector | MIT, UT Austin & Univ of Cinc

Banani Mohapatra

Senior Manager, AI/ML & Data Science at Walmart | Generative AI,LLM | IIT Delhi

Cohort 1

Cohort 2

Users are bombarded with AI-powered products every day, but most can’t tell which ones actually work.In a market flooded with AI marketing claims, trust and proof are what set great products apart. To stand out, you need data to back up your product’s credibility, not just a clever demo.That is where AI Evaluations and Analytics come in. 47% of organizations have already faced AI gone wrong (McKinsey, 2025). Even leaders like OpenAI and Anthropic are self-insuring because traditional coverage cannot keep up.AI systems are unpredictable. The real moat today is how well you evaluate and iteratively improve them. 👉 About This Course You don’t need another coding course.This is a methodology and framework course for product and data leaders shaping the next generation of AI products. Learn a battle-tested framework to evaluate, monitor, analyze and improve your AI products — from testing first prototype to monitoring and iterating in production.Make AI decisions backed by evidence, not intuition.You’ll apply the AI Evals &amp; Analytics Playbook — a proven system that combines human and automated tools for both scalability and reliability. Through real-world examples, exercises, and demos, you’ll learn how to:✅ Design evaluation workflows that align with your product across different development stages✅ Set up pre-release evaluations and assessments to enable faster, data-driven iteration✅ Build post-release monitoring and analytics that keep your AI systems reliable and accountable By the end, you’ll walk away with an actionable, organization-ready evaluations and analytics plan for your own AI product. 🗓️ Course Plan This accelerated program condenses four weeks of content into two for a fast, immersive learning experience. Lesson 1: AI Evaluations - Why &amp; What- Getting buy-in on AI Evals- Build your first AI Evals team- Differentiate Eval, Analytics, and XAI- Integrate AI evaluation into your product lifecycleExercise: Section 1 of your playbook Lesson 2: How to Evaluate AI Products- AI Evals &amp; Analytics Framework Overview- Human evaluation vs. Automated evaluation- Use LLM-as-a-judge- Design rubrics and metricsExercise: Section 2 of your playbook Lesson 3: Experiment Design and AI Eval Tools- Evaluator model selection- Curate or generate test sets- Experiment design- Overview of popular eval toolsExercise: Sections 2 &amp; 3 of your playbook Lesson 4: Monitoring &amp; Analytics- Handle user data- Product monitoring practices for AI products- Understand leading &amp; lagging indicators- Conduct post-launch review and analyticsExercise: Sections 3 &amp; 4 of your playbook 🎁 Founding Cohort Benefits (Oct 27 - Nov 9) Join our first cohort and help shape the course. Receive personalized guidance, in-depth discussions on your evaluation challenges, and access to our professional community of product leaders, data practitioners, and AI builders.

Evals & Analytics: The New Core Moat of Great AI Products

Intensive sessions with frameworks, tactics, and exercises.

Live Interactive Lectures + Workshops
 

Product LeadersBuilding AI products and need a clear playbook for evaluation frameworks, success metrics, and shipping with confidence.

Data LeadersRedefining team scope and structure in the AI era, aligning evaluation, analytics, and accountability.

Seasoned Data ScientistsLeveling up with AI product skills: learn AI evals, LLM-as-a-judge, and design AI-specific performance metrics.

Who is this course for

4 sessions (2 hours each) over 2 weeks. Build your first AI Evals and Analytics playbook for your current product use case.

Walk Away with Execution-ready Playbook

Create clear ownership: who writes rubrics, validates metrics, and holds veto power. Keep Legal, Trust &amp; Safety, and SMEs aligned without evaluation becoming a bottleneck.

Build Your First AI Evals Team

Run sniff tests, build quantitative evaluation pipelines, and design experiments that prove your AI beats the baseline. Know when to use human labels vs. LLM-as-a-judge.

Execute Comprehensive Pre-launch Testing

Handle stochastic outputs and subjective quality with proper experiment design. Choose the right methodology, set sample sizes, and define guardrails that protect business metrics.

Design Experiments for AI products

Set up leading indicators (retry rates, confidence scores) and lagging metrics (CSAT, cost). Build escalation procedures and run structured post-launch reviews at 15, 30, and 60 days.

Monitor Product and Catch Issues Early

Build trust and proof for your AI product through pre-release testing, in-production monitoring, and analytics.

Shopify

Carvana

Harvard University

Arizona State University

Stella Liu

Head of Applied Scientist, AI Evals

System1

Uptake

Seasalt.ai

UCLA

University of Washington

Amy Chen

Bad data is everywhere - causing broken dashboards, misleading insights, and costly business mistakes. But data quality isn't just about cleaning up messes; it's about preventing them before they happen. In this course, you’ll learn how to systematically prevent, diagnose, and resolve data quality issues. Whether you're constantly fixing reports, debugging pipelines, or advocating for data governance, this course gives you the tools to take control of your data and ensure its reliability. What You’ll Learn:✅ Identify &amp; Prevent Bad Data – Analyze the root causes of data issues and design proactive solutions to stop them before they spread.✅ Diagnose &amp; Debug with SQL – Develop a structured approach to finding and fixing data anomalies using SQL and debugging best practices.✅ Automate Quality Checks – Build automated alerts and implement best practices to maintain reliable, high-quality datasets.✅ Secure Stakeholder Buy-In – Learn how to make a compelling case for prioritizing data quality and get leadership support for governance initiatives. By the End of This Course, You’ll Be Able To:🚀 Assess the health of your data and pinpoint weak spots in your data lifecycle.🔍 Investigate &amp; Fix data quality issues with confidence, reducing time spent firefighting.⚙️ Implement Automation to catch and prevent errors before they impact decision-making.🤝 Advocate for Change and drive a culture of data quality in your team and company. This course is designed for data analysts, data scientists, and data engineers who want to level up their foundational data skills and ensure their insights are built on trustworthy, high-quality data. If you’re ready to stop firefighting bad data and start preventing it, this course is for you.

Transform messy data into reliable insights with proven data quality strategies

The course opens for pre-reads and a self-reflection homework assignment.

4 days before the 1st session

The required live sessions will be held across two weekends. Both days will cover 4 modules on each day, with breaks in between. Each day will end with a live Q&amp;A session.

Saturdays

Both weeks include a project, and there is a short homework assignment prior to the first live session. While the projects &amp; homework are not&nbsp;required, completing them will greatly improve the value you can get out of our time together.

Weekly projects

Each learner can sign up for a 15-minute slot with the instructor to get personalized suggestions at the end of the course.

Office Hours w/ Shailvi

Data Analysts &amp; Data Scientists who want to prevent bad data instead of constantly fixing errors and dealing with misleading insights.

Data Engineers who want to build reliable pipelines with automated quality checks to ensure a trustworthy, high-integrity data environment.

Data Practitioners who need to secure company-wide buy-in for data governance initiatives that drive better business decisions.

How This Course Will Elevate Your Data Quality Skills

Evaluate data lifecycle risks and formulate prevention techniques to mitigate errors before they escalate. Construct frameworks for ensuring clean, reliable datasets that enhance analytical accuracy and decision-making.

📊 Design Proactive Strategies to Eliminate Data Quality Issues

Analyze data anomalies, trace data lineage, and synthesize debugging techniques using SQL. Develop a structured, repeatable approach to resolving data quality issues efficiently, reducing downtime and improving trust in your datasets.

🔍 Diagnose and Debug Complex Data Issues with Precision

Engineer robust validation mechanisms and deploy automated alerts to monitor data integrity. Optimize pipeline reliability by applying coding best practices that enhance efficiency and minimize manual intervention.

⚙️ Implement Automated Quality Checks to Strengthen Pipelines

Justify the business impact of poor data and construct persuasive arguments for data governance investment. Lead cross-functional discussions that drive alignment, fostering a culture of proactive data quality management.

🤝 Influence Stakeholders to Prioritize Data Quality Initiatives

Boost your data quality skills! Prevent, diagnose & fix bad data to drive accuracy 🎯 efficiency ⚙️ and better business decisions 💵