Free Lesson
Experimentation in the AI Era: Lessons from the Trenches
Part of The AI Evaluation Handbook
60 min
Jul 3, 2026 1:30 PM
Virtual (Zoom)
In this video
What you'll learn
Why traditional A/B testing falls short
See how GenAI breaks traditional testing frameworks through spillage, SUTVA failures, and constantly evolving models.
How to measure success in AI-powered experiences
Learn practical frameworks to design metrics and measure success even when AI treatments and feedback loops change.
The future of experimentation: Adaptive & autonomous systems
Eliminate testing bottlenecks using bandits, AI-driven design, and continuous closed-loop learning.
Why this topic matters
GenAI accelerates code, but traditional testing creates a massive bottleneck. When model predictions constantly shape user behavior, and user feedback loops alter the model, static frameworks break down. Teams that can't adapt risk shipping blind. We'll explore why old methods fail and show you how to transition to the future of high-velocity, adaptive, and autonomous experimentation.
