Jazgul Ismailova

Founder and CPO at Glimma AI

AI-Moderated Website Testing

Understand its strengths, limitations, and how it differs from traditional moderated and unmoderated website testing.

Understand when and how to use AI-moderated website testing.

Learn how to define objectives, create realistic tasks, and write instructions that encourage useful user feedback.

Design and set up your own AI-moderated website study.

Learn how to combine user feedback to identify friction, understand why it occurs, and prioritise improvements.

Turn user feedback into actionable UX improvements.

Learn from a world-leading expert how to design and analyze trustworthy A/B tests to evaluate ideas, integrate AI/ML, and grow your business

14 Sept 2026

Author of best-selling A/B testing book

Microsoft

Airbnb

Amazon

Dr. Ronny Kohavi

Technical fellow and VP at Microsoft & Airbnb, co-author of A/B testing book

1 June 2026

9 Nov 2026

Through multiple real examples of well-run experiments and stories from Microsoft, Amazon, and Airbnb, you will see the humbling reality that we are terrible at assessing the value of ideas.Trivial changes can be surprisingly useful, whereas large efforts often fail. Accelerate innovation by building Minimum Viable Products (MVPs) and features, and make the organization more evidence-based and humble as it adopts and learns to use evidence from the gold standard in science—the controlled experiment.You will understand the challenges in designing and running trustworthy randomized controlled experiments (A/B tests), including the importance of the Overall Evaluation Criterion (OEC), scaling, common pitfalls, and Twyman's Law. We cover cultural challenges, institutional memory, maturity model, observational causal studies, offline evaluation, AI/Machine learning and triggering, build-vs-buy trade-offs, challenges, and requested topics. The Advanced Topics course is designed as a technical follow-on: <a target="_blank" rel="noopener noreferrer nofollow" href="https://bit.ly/AdvancedABFS">https://bit.ly/AdvancedABFS</a>

Empower your organization to be data-driven and innovative

Internalize the humbling reality: we are poor at assessing the value of ideas

Understand the key advantages and limitations of A/B testing

Learn how to design metrics and the Overall Evaluation Criterion

Learn how to design trustworthy A/B tests

Learn about the cultural challenges

Learn about the relationship to AI and Machine Learning Modeling

Data science managers and scientists will be able to design and interpret the experimental results in a trustworthy manner

Program managers focused on growth, revenue, conversions, and prioritization will understand how to provide the org with robust clear metric

Engineering leaders will be able to make organizations more data-driven and efficient with fewer severe incidents through A/B tests

Gain hands-on mastery of AI research tools and frameworks to deliver trusted insights faster while maintaining the rigor that matters.

September 2026

AI Research Expert | Johns Hopkins PhD

Google

Capital One

Bloomberg

Pinterest

John Whalen, PhD

PhD. O'Reilly author. Host of AI for UX, where I interview AI tool founders.

July 2026

The researchers I talk to feel two things at once: overwhelmed by the pace of AI, and watching real possibilities open up that didn't exist two years ago.Both are true. AI lets you run 500 interviews across 12 countries in a week. It surfaces patterns across thousands of sessions. It opens research to teams who never had budget for it. Synthetic users let you pressure-test ideas in hours. These are new ways of working.But how do you trust the output? Which tools deliver, and which produce confident nonsense?That's the shift this course is built around: from doing research one study at a time to orchestrating systems that scale. You'll get hands-on with the platforms researchers actually use (Listen Labs, Coloop, Marvin, Synthetic Users), hear from the founders building them, and build the judgment to know what holds up.This is Course 1 of 2 in the Certified Agentic Research Specialist program — your foundation. Course 2, Agentic AI for Research, builds on it. The researchers pulling ahead are orchestrators — conducting research with more reach, impact, and confidence than before.No hype. Just what works.

Create Customer Insights with AI You Can Actually Trust

Conduct AI-moderated customer interviews confidently

Extract insights from 50+ interviews using AI-powered analysis tools

Develop frameworks to assess new AI tools and know when they actually work

Use synthetic users for rapid testing and validation of ideas before launch

Guide your team or organization in adopting AI research tools and best practices

Connect with practitioners navigating AI research together and sharing insights

You've worked with customer insights in some capacity—whether you've conducted research yourself, collaborated with researchers on your team

Experience with customer research or user feedback

This is a practical course with weekly exercises using AI platforms. You'll need time to experiment and apply what you learn.

Willingness to try new tools hands-on

We use tools and prompting techniques that don't require any coding knowledge. If you can write clear instructions, you're ready.

No technical background required

Researchers seeking to deliver insights faster at scale using AI—while maintaining the quality and rigor that makes their work trustworthy.

PMs and research leaders balancing the need for rapid customer insights with concerns about maintaining quality, rigor, and team standards.

Designers seeking practical AI tools to understand users, test concepts, and make evidence-based design decisions with confidence and speed.

Build and validate an AI Product, from scratch.

Founder | CPO | 10 years in Product

Finastra

Trading Central

Printify

Madalina Turlea

Product leader with 10+ years experience in FinTech | Founder &CPO Lovelaice

2x founder | CTO | 14 years in Tech

Teleclinic

Nilo

SIXT Group

Freeletics

Catalina Turlea

Founder & CEO @Lovelaice | Co-founder & ex-CTO @ nilo | 14 years in tech

Civicthon

September

<blockquote>Built from 1,500+ real AI experiments · 5.0 rating Taught by a product leader and a former CTO, using the same evaluation process we apply to our own product and customer work. Your AI feature returns polished answers. But can you prove they are accurate, consistent, and good enough to ship?Testing two or three examples is not enough. Generic “helpfulness” scores will not catch the failures that matter. And waiting for customer feedback means discovering the problem after trust is already lost.In this hands-on workshop, you’ll build and evaluate a real AI feature from start to finish.You’ll run the same prompt across multiple models, review outputs blind, uncover the failures the AI actually makes, turn those failures into targeted evals, improve the prompt, and measure whether the new iterations performs better.By the end, you will have:<ul><li>A tested production-style system prompt</li><li>A quality baseline across multiple AI models</li><li>An annotated set of real AI outputs</li><li>A prioritized map of the feature’s failure modes</li><li>A repeatable process you can use on your next AI feature</li></ul>You will perform the evaluation yourself, make product decisions from the results, and leave with a working system you understand.</blockquote>

Build the skills to evaluate AI features, improve them with evidence, and make confident ship decisions, without relying on vibe checks.

Build a production-ready AI prompt

Find and prioritize your AI’s real failure modes

Build an evaluation suite to scale your testing

Product managers and product leads responsible for an AI-powered feature

Founders moving an AI product from prototype toward production

Domain experts who need to define what a correct AI answer looks like

Eliminate uncertainty in shipping AI features

GenAI Product Builder @ Google, ex-Meta

Marily Nika, Ph.D AI/ML

Senior Data, ML and GenAI Scientist

George Zoto

Senior NLP/AI Scientist

August-2026

“Does it work?”... “Is it good enough?”... “Can we ship it?”...How do you answer these questions for AI products? You’re responsible for “running evals” but what does that mean?How do you choose the right metrics, interpret fuzzy results, and make a confident decision?This course gives you a framework to do just that.<ul><li>Map user value to evaluation (eval) objectives so your metrics aren’t abstract. Define success then translate it into measurable criteria.</li><li>Choose metrics you can actually maintain: capability, safety, UX friction, latency, cost and “does this reduce support tickets or increase activation.”</li><li>Set ship/no-ship thresholds you can defend to leadership.</li><li>Build lightweight workflows that work in real teams: human review where it matters, automation where it lasts, documentation that drives decisions.</li><li>Consider domain constraints (e.g., healthcare safety) and know what to avoid: silent failures, misleading proxy metrics and tests that don’t reflect production.</li><li>Tie everything to ROI: impact vs unit cost, eval coverage vs reliability, and the minimum viable monitoring you need post-launch.</li></ul>Experience AI evals through a case-based approach with a real AI product that we evaluate together.

Acquire and develop a critical skill for product managers who are leading and contributing to AI products.

Make confident ship or hold decisions for AI features

Translate user value into clear evaluation goals

Choose the right metrics for capability, safety, UX, and cost

Set defensible thresholds leadership will trust

Build lightweight evaluation workflows teams can maintain

Navigate domain constraints and avoid common failures

PMs leading AI features, growth, or platform initiatives

PMs who partner with ML teams and want to set evaluation standards

PMs who need to make clear “ship or hold” calls without doing the engineering

Agentic AI

Coding with AI

AI Workflows

Claude Code

OpenClaw

Vibe Coding

AI Evals

AI Transformation

RAG & Search

AI for PMs

AI for Engineers

AI for Designers

AI for Marketers

AI for Founders

Product Sense

Product Discovery

User Research

Prototyping

Growth

Analytics

Tech Foundations

Strategy

Influence

Leadership

Career Growth

Machine Learning

LLM Ops

Context Eng

Security

System Design

Figma

Design Systems

Visual Design

Design Strategy

Automation

Content Marketing

Demand Gen

Go-to-Market

Product Marketing

Positioning

Social Media

Brand

B2B Marketing

SEO & AEO

AI for Leaders

AI Governance

Communication

Management

People Operations

Exec Presence

Storytelling

Goal-setting

Personal Brand

Fundraising

Everyone

Operators

Data Scientists

Business Analysts

User Researchers

Customer Success

Project Managers

HR Professionals

Sales People

Lawyers

Finance

Investors

Real Estate

Educators

Creators

Product managers who drive strategy, development, and launch of products.

Designers who craft user experiences, interfaces, and visual design.

User researchers conducting interviews, usability studies, and customer research to inform product decisions.

Applying AI tools and techniques to professional workflows — assign alongside more specific AI topics.

User experience design — information architecture, interaction design, usability, and end-to-end experience flows.

Conducting user interviews, surveys, and usability studies to understand customer needs and validate decisions.

AI-Moderated Website Testing

What you'll learn

Why this topic matters

You'll learn from

Jazgul Ismailova