Director of AI Education at A+ Active

Cityblock Health

HashiCorp

The New School

PrismML

Webflow

Jai Bhagat

Safely Switching AI Models

Identify the outputs, workflows, and user expectations that need regression checks before a switch.

Define what must stay stable

Design a small evaluation set that reflects real prompts instead of relying only on vendor benchmarks.

Compare models on your tasks

Plan a staged switch with monitoring, fallbacks, and clear criteria for rollback.

Roll out with guardrails

Eliminate uncertainty in shipping AI features

GenAI Product Builder @ Google, ex-Meta

Marily Nika, Ph.D AI/ML

Senior Data, ML and GenAI Scientist

George Zoto

Senior NLP/AI Scientist

August-2026

“Does it work?”... “Is it good enough?”... “Can we ship it?”...How do you answer these questions for AI products? You’re responsible for “running evals” but what does that mean?How do you choose the right metrics, interpret fuzzy results, and make a confident decision?This course gives you a framework to do just that.<ul><li>Map user value to evaluation (eval) objectives so your metrics aren’t abstract. Define success then translate it into measurable criteria.</li><li>Choose metrics you can actually maintain: capability, safety, UX friction, latency, cost and “does this reduce support tickets or increase activation.”</li><li>Set ship/no-ship thresholds you can defend to leadership.</li><li>Build lightweight workflows that work in real teams: human review where it matters, automation where it lasts, documentation that drives decisions.</li><li>Consider domain constraints (e.g., healthcare safety) and know what to avoid: silent failures, misleading proxy metrics and tests that don’t reflect production.</li><li>Tie everything to ROI: impact vs unit cost, eval coverage vs reliability, and the minimum viable monitoring you need post-launch.</li></ul>Experience AI evals through a case-based approach with a real AI product that we evaluate together.

Acquire and develop a critical skill for product managers who are leading and contributing to AI products.

Make confident ship or hold decisions for AI features

Translate user value into clear evaluation goals

Choose the right metrics for capability, safety, UX, and cost

Set defensible thresholds leadership will trust

Build lightweight evaluation workflows teams can maintain

Navigate domain constraints and avoid common failures

PMs leading AI features, growth, or platform initiatives

PMs who partner with ML teams and want to set evaluation standards

PMs who need to make clear “ship or hold” calls without doing the engineering

Shift from cost-cutting process automation to high-ROI AI-human augmentation by uncovering hidden cognitive capacity, mitigating enterprise 

Workshop Lead: Netflix, Google, LinkedIn

Google

Netflix

Microsoft

LinkedIn

Paul Bryan

I help organizations design and develop AI products that deliver business value.

Cohort 1

Cohort 2 is in waiting list mode

<blockquote>There is a tacit assumption that automation means getting rid of humans and letting AI do all the work. Leaders with a replacement mentality view workflows as the mechanical process steps in a flowchart. The risk is that this form of automation liquidates the implicit human expertise, judgment, and domain context that make an organization successful. Worse, when every competitor adopts the same AI models for the same incremental efficiencies, it creates a race to the bottom that commoditizes your core offerings.This course offers a strategic alternative: Cognitive Task Analysis (CTA). Derived from high-stakes human factors engineering, CTA makes invisible cognitive work visible, quantifies human pattern matching, and establishes where AI should absorb operational friction so human talent can focus on high-margin, value-compounding initiatives.Instead of defending against headcount cuts, you will learn to map out a clear, high-ROI augmentation roadmap for 2027 that protects core human expertise, mitigates implementation risks, and proves to leadership how retaining and multiplying staff impact drives top-line growth.</blockquote>

Deploy AI for Efficiency, Employ Humans for Efficacy

Conduct Cognitive Task Analysis (CTA)

Map Systemic Friction & Quality Risks

Architect Human-AI Service Blueprints

Design Non-Commoditizable Moats

Build Risk-Adjusted ROI Business Cases

Pretype & Execute a 6-Week Charter

Product and business leaders who need to capture AI operational efficiencies without losing key talent or core organizational expertise.

AI strategists and innovation leads looking for rigorous, risk-adjusted frameworks to prove AI can drive top-line growth.

Practitioners who want to give leadership a value-compounding AI roadmap that fully leverages human strengths and expands capacity.

Bring one real workflow and leave with clearer roles for AI and people to test 

Human-AI Work & Systems Advisor

Joyce Ventures

University of Wisconsin-Madison

Center on PBIS

Joyce White Nelson 

Systems-Change Advisor | Human Judgment & AI-Assisted Work | Founder, iQuoteHero

Founding Live Workflow Lab — September 2026

AI can help you move faster and sound 'good'. But if the workflow is unclear, speed and false positives can also create rework, weak review, lost context, generic output, or confusion about who owns the result.In this working Lab, you will bring one real AI-assisted workflow, decision, client deliverable, communication, or recurring task. You will map how the work currently happens, identify what AI should assist with, decide what needs human judgment and verification, and redesign with clearer roles for you, AI, and anyone else involved.Send me your workflow, decision, or use case ahead of time. That helps me shape examples and guidance around real situations.You will leave with:<ul><li>A completed Human + AI Workflow Map.</li><li>Clear decisions about what to delegate, retain, verify, or change.</li><li>Defined accountability and judgment points.</li><li>One practical experiment to test in your work.</li><li>A reusable method for evaluating future AI-assisted workflows.</li></ul>After the workshop, I will follow up by email or brief review call to help you examine what happened when you tested the workflow. We will look at what improved, what created friction, and what to keep, revise, or stop.

Redesign one real AI-assisted workflow so the Human/AI roles are defined and make  human judgment, accountability, or expertise visible.

 Map one workflow as it works today

Decide where AI belongs

. Protect human judgment and accountability

 Identify the most important risk or rework point

 Design one improved workflow experiment

Workflow to measure the outcome 

Experienced entrepreneurs and founders Already using AI and want clearer judgment, better workflows, and less hidden rework.

Consultants, coaches, and advisors Use AI in client work but want to protect quality, credibility, voice, and professional responsibility.

Leaders, educators, and team facilitators Need ways to decide where AI helps, where humans stay accountable, and people keep learning.

A practical workshop on evaluating AI agents before launch

Founder & CEO | Ex-Adobe, WF, eBay | HBS

Adobe

Wells Fargo

eBay

Ganesan Anand

Founder, Formerly AI Product & Tech Leader (Adobe, Wells Fargo, eBay) · HBS alum

Next Cohort

<blockquote>Move your AI features from a vibe check to a value check. “Did the agent do the right thing? Would I catch it if it didn’t?” Every team shipping AI features is asking some version of these questions without a reliable way to answer them. An agent doesn’t just say the wrong thing. It does the wrong thing, and teams often find out from a user.Typical AI training falls into two traps: abstract theory that never leaves the slide deck, or disconnected techniques that stop short of a production-ready system. This workshop bridges that gap. I build enterprise infrastructure for AI evaluation every day, and I’ve seen that real adoption only happens when teams can trust their AI to perform consistently.You’ll learn a repeatable method for applying Responsible AI in practice, with a focus on reliability and safety. You’ll learn which metrics matter for your product, identify the edge cases that undermine trust, define how the product should respond, and prove it works before customers see a problem.</blockquote>

Learn a repeatable evaluation framework for building and releasing reliable AI features.

Know exactly what to measure

Find failures before your users do

Get a pass/fail read before release

Keep quality from slipping after launch

Access to leading commercial and open-source AI models is provided during the session for the eval exercises. Nothing to set up or pay for.

Model access is provided

Come ready to dig into how AI features fail and how to catch it. That mindset is all it takes.

Curiosity about AI quality

The session is live and remote, so a solid connection matters. Everything runs in-browser, and the no-code evaluation platform is provided.

Stable internet

AI engineers and product managers building, evaluating, or preparing to release AI features

Founders turning an AI prototype into a reliable product

Engineering and product leaders accountable for AI quality, safety, and release decisions

Agentic AI

Coding with AI

AI Workflows

Claude Code

OpenClaw

Vibe Coding

AI Evals

AI Transformation

RAG & Search

AI for PMs

AI for Engineers

AI for Designers

AI for Marketers

AI for Founders

Product Sense

Product Discovery

User Research

Prototyping

Growth

Analytics

Tech Foundations

Strategy

Influence

Leadership

Career Growth

Machine Learning

LLM Ops

Context Eng

Security

System Design

Figma

Design Systems

Visual Design

Design Strategy

Automation

Content Marketing

Demand Gen

Go-to-Market

Product Marketing

Positioning

Social Media

Brand

B2B Marketing

SEO & AEO

AI for Leaders

AI Governance

Communication

Management

People Operations

Exec Presence

Storytelling

Goal-setting

Personal Brand

Fundraising

Everyone

Operators

Data Scientists

Business Analysts

User Researchers

Customer Success

Project Managers

HR Professionals

Sales People

Lawyers

Finance

Investors

Real Estate

Educators

Creators

Product managers who drive strategy, development, and launch of products.

Engineers and engineering leaders who build, deploy, and scale software systems.

Data scientists who apply statistical and ML techniques to extract insights from data.

Applying AI tools and techniques to professional workflows — assign alongside more specific AI topics.

Measuring and evaluating AI model performance, quality, and reliability for production use cases.

Operating and deploying large language models in production — fine-tuning, inference, monitoring, and cost optimization.

Safely Switching AI Models

What you'll learn

Why this topic matters

You'll learn from

Jai Bhagat