AI Evaluation: From Evidence to Decision
Dave Rogers
Research · Data · Program Leadership
You don't build the AI. You decide what happens next.
A practical, live workshop for AI program managers, technical project managers, product leaders, and Chiefs of Staff who need to assess AI performance claims, challenge the evidence, and recommend what to do next — without needing to build or train a model.
Learn to distinguish convincing numbers from trustworthy evidence and turn evaluation results into decisions about what to ship, change, investigate, or stop.
What you’ll learn
Learn to ask better questions, recognize trustworthy AI evidence, and make confident, evidence-based decisions about what happens next.
Connect AI objectives to evaluation methods and real-world outcomes.
Spot when a metric measures the wrong thing.
Understand how rubrics, calibration, and evaluator agreement affect reliability.
Recognize when evaluation results need more scrutiny.
Look beyond “92% accurate” to uncover hidden failures, edge cases, and bias.
Identify gaps between headline results and real-world performance.
Set decision thresholds before results are known.
Decide whether to ship, pivot, narrow scope, gather more evidence, or stop.
Understand key terms like ground truth, eval sets, drift, and LLM-as-judge.
Follow technical discussions and ask informed questions without needing to build the AI.
Use 18 ready-to-use questions in plain-language and advanced versions.
Challenge assumptions, surface evidence gaps, and contribute to better decisions.
Workshop agenda
Introduction: Why AI Evaluation Is Hard (10 min)
A live vote reveals how quickly people can disagree about what “good” means. Meet Kai, a household robot learning to put away groceries, whose evaluation journey runs throughout the workshop.
Part 1: What Are We Measuring? (30 min)
Explore the AI Learning Loop, from objectives and ground truth to evaluation, deployment, and learning from failure.
Part 2: Can We Trust the Measurement? (35 min)
Explore rubrics, calibration, and inter-rater reliability. See how improving the evaluation process can improve agreement between human evaluators.
Part 3: Does the Evidence Support the Decision? (35 min)
Kai is “92% accurate.” Investigate what that headline number hides across segments, edge cases, sampling, and bias. Decide what the evidence actually justifies.
Apply & Close (10 min)
Bring the frameworks and 18 practical questions into your next AI evaluation conversation. Identify what you would ask, what evidence you would need, and what decision you would recommend.
Learn directly from Dave
Dave Rogers
Research & analytics leader helping teams make better AI decisions
Who this workshop is for
Product, program, and technical program managers who review AI results they didn't produce and need to decide what happens next.
Chiefs of Staff, operations, and business leaders asked to approve, fund, or expand an AI pilot.
Data, analytics, research and insights professionals who assess AI performance and turn findings into recommendations for teams and leaders.
What's included
Live sessions
Learn directly from Dave Rogers in a real-time, interactive format.
Live, expert-led 2-hour workshop
Research-grade frameworks applied through guided cases, live votes, and hands-on exercises.
AI Evaluation Vocabulary Guide — Complete Edition ($29 value)
100+ terms in plain English: what each means, why it matters, where you’ll hear it, and where teams get it wrong.
AI Evaluation Evidence Pack — Complete Edition ($39 value)
A field guide to the AI Learning Loop, reliability, validity, and real-world fit, with six professional perspectives.
Follow-up live Q&A
30 minutes on Monday, November 16, 10:00–10:30 AM PST. Bring the questions that come up as you apply the frameworks.
Maven Guarantee
Your purchase is backed by the Maven Guarantee.
Frequently asked questions
Maven for Teams
Reimbursement
Get your company to pay
Everything L&D needs: email template, receipts, and certificate of completion.
Get reimbursedTeam discount
Learn with your teammates
Save 20%+ when 2 or more teammates enroll in the same cohort.
Save 20%+ with a teamPrivate cohort
Run a cohort for your org
A dedicated cohort with a custom schedule and curriculum, tailored to your team.
Book a private cohort$299
USD
8–10pm EST