Simple AI Labs

Stop Shipping AI Agents on Vibes

Uber

Adobe Marketo Engage

Simplified

Arizona State University

KD Deshpande

AI Founder, Building the Future of AI Agents & Agentic Workflows.

Why "it passed our tests" isn't enough; the PM/Legal/Ops trust gap (origin story).

The vibes problem

What an eval actually is: input → expected behavior → judge. The 4 kinds that matter: correctness, quality, compliance

The mental model

Catch agent failures before your users (or your customers) do.  Leave with a reusable framework.

Hands-on

Build a real AI agent, find where it breaks, and improve it with evals you can trust, working the full loop hands-on.

ML Engineer with 25 years of experience

Airbnb

GitHub

DataRobot

AlixPartners

Hamel Husain

ML Engineer with 20+ years of experience.

ML Systems & Applied AI Evals Researcher

Google

UC Berkeley

Stanford University

Shreya Shankar

ML Systems Researcher Making AI Evaluation Work in Practice

Cohort 5

Cohort 6

Save 25% ($1,050) on September's cohort <a target="_blank" rel="noopener noreferrer nofollow" href="https://maven.com/parlance-labs/evals/6/join?promoCode=LOOK-AT-YOUR-DATA">with this link</a>.--- 🏆 The most comprehensive AI evals course available. We've refined this material with over 4,500 engineers and PMs from teams like OpenAI, Google, Meta, Amazon, and Microsoft. <a target="_blank" rel="noopener noreferrer nofollow" href="https://testimonial.to/ai-evals-course/all">Read what students say →</a>📚 View <a target="_blank" rel="noopener noreferrer nofollow" href="https://docs.google.com/document/d/1KSUlzN-C8z1q_gHduCi6eKgCzlHVV6u3dT0aypqi_YI/edit?usp=sharing">the syllabus here </a>More than a course: an ongoing system, tools, and community for shipping better AI.<ul><li>🎮 A private Discord community for ongoing support, even after the course.</li><li>🤖 6 months of access to our AI Evals assistant.</li><li>♾️ Lifetime access to all recordings, materials, and future cohorts.</li><li>💬 10+ hours of live office hours to get your questions answered.</li></ul>Do you catch yourself asking any of these while building AI applications?<ol><li>How do I test outputs that need subjective judgment?</li><li>If I change the prompt, how do I know I am not breaking something else?</li><li>Where should I focus my efforts? Do I need to test everything?</li><li>What if I have no data or customers? Where do I start?</li><li>What should I measure, and what tools should I use?</li><li>Can I automate evals, and how do I trust it?</li></ol>If so, this is for you. All sessions are live and recorded.📝 We've completely refreshed the material for our Sep 2026 cohort. Scroll down to see the syllabus.

 Build, evaluate, and improve AI agents that work in production.

Build an agent you can measure

Find what's broken with error analysis

Build evaluators you trust

Catch regressions with CI/CD

Red-team for safety before users do

Improve accuracy and cut cost with evidence

Engineers and PMs who ship prompt changes and hope nothing breaks. (You'll learn to measure impact before and after every change.)

Teams still spot-checking AI outputs by hand instead of measuring systematically. (You'll learn how to build automated evals you can trust.)

Leaders who don't know where their AI is failing or where to invest resources. You'll learn how to systematically find &amp; prioritize issues.

Build AI Agent Skills that Actually Work - 3 Hours Hands-on Workshop

Product Leader and Super IC at Google

Disney+ Hotstar

Zynga

Anshumani Ruddra

Product Super IC at Google with 22+ years crafting experiences and blockbuster products

Cohort 07

Cohort 08

Most product builders use AI agents the same way they used search - typing queries and hoping for the best. That's leaving most of the value on the table.The teams pulling ahead aren't using better prompts. They're teaching their agents. Agent Skills are reusable, testable instruction sets that tell AI agents exactly how to handle specific workflows: your deployment process, your code review standards, your team's edge cases. Write one skill well, and every agent on your team benefits from it, every time, automatically.The problem: nobody is teaching product builders how to create them. Most documentation is written for a narrow technical audience, and the tooling is fragmented across Claude Code, Codex, and Gemini.In this hands-on workshop, you'll go from zero to a working skill installed in your own agent: one that reflects your real workflows, not generic templates. You'll also learn how to evaluate and improve skills systematically, so they get better over time instead of drifting.If you build or are hoping to build products with AI agents: this is the highest-leverage skill you can develop right now.

🤖 Learn to build procedural Agent Skills, transforming your professional mindset from solo prompter to manager of elite AI agents

Shift to an AI Management Mindset

Build Custom, Reusable Agent Skills

Leverage the Open-Source Ecosystem

Evaluate and Iterate for 10x Output

Product Managers: who use Claude Code or Codex daily but rely on ad-hoc prompts - and want repeatable, team-shareable AI workflows.

Entrepreneurs &amp; Founders: building AI-native products who need their agents to encode business logic, not just generate code on demand.

Tech Executives &amp; Product Leaders: who want their teams to stop reinventing prompts and start shipping a shared skills library.

Eliminate uncertainty in shipping AI features

GenAI Product Builder @ Google, ex-Meta

Marily Nika, Ph.D AI/ML

Senior Data, ML and GenAI Scientist

George Zoto

Senior NLP/AI Scientist

August-2026

“Does it work?”... “Is it good enough?”... “Can we ship it?”...How do you answer these questions for AI products? You’re responsible for “running evals” but what does that mean?How do you choose the right metrics, interpret fuzzy results, and make a confident decision?This course gives you a framework to do just that.<ul><li>Map user value to evaluation (eval) objectives so your metrics aren’t abstract. Define success then translate it into measurable criteria.</li><li>Choose metrics you can actually maintain: capability, safety, UX friction, latency, cost and “does this reduce support tickets or increase activation.”</li><li>Set ship/no-ship thresholds you can defend to leadership.</li><li>Build lightweight workflows that work in real teams: human review where it matters, automation where it lasts, documentation that drives decisions.</li><li>Consider domain constraints (e.g., healthcare safety) and know what to avoid: silent failures, misleading proxy metrics and tests that don’t reflect production.</li><li>Tie everything to ROI: impact vs unit cost, eval coverage vs reliability, and the minimum viable monitoring you need post-launch.</li></ul>Experience AI evals through a case-based approach with a real AI product that we evaluate together.

Acquire and develop a critical skill for product managers who are leading and contributing to AI products.

Make confident ship or hold decisions for AI features

Translate user value into clear evaluation goals

Choose the right metrics for capability, safety, UX, and cost

Set defensible thresholds leadership will trust

Build lightweight evaluation workflows teams can maintain

Navigate domain constraints and avoid common failures

PMs leading AI features, growth, or platform initiatives

PMs who partner with ML teams and want to set evaluation standards

PMs who need to make clear “ship or hold” calls without doing the engineering

A practical workshop on evaluating AI agents before launch

Founder & CEO | Ex-Adobe, WF, eBay | HBS

Adobe

Wells Fargo

eBay

Ganesan Anand

Founder, Formerly AI Product & Tech Leader (Adobe, Wells Fargo, eBay) · HBS alum

Next Cohort

<blockquote>Move your AI features from a vibe check to a value check. “Did the agent do the right thing? Would I catch it if it didn’t?” Every team shipping AI features is asking some version of these questions without a reliable way to answer them. An agent doesn’t just say the wrong thing. It does the wrong thing, and teams often find out from a user.Typical AI training falls into two traps: abstract theory that never leaves the slide deck, or disconnected techniques that stop short of a production-ready system. This workshop bridges that gap. I build enterprise infrastructure for AI evaluation every day, and I’ve seen that real adoption only happens when teams can trust their AI to perform consistently.You’ll learn a repeatable method for applying Responsible AI in practice, with a focus on reliability and safety. You’ll learn which metrics matter for your product, identify the edge cases that undermine trust, define how the product should respond, and prove it works before customers see a problem.</blockquote>

Learn a repeatable evaluation framework for building and releasing reliable AI features.

Know exactly what to measure

Find failures before your users do

Get a pass/fail read before release

Keep quality from slipping after launch

Access to leading commercial and open-source AI models is provided during the session for the eval exercises. Nothing to set up or pay for.

Model access is provided

Come ready to dig into how AI features fail and how to catch it. That mindset is all it takes.

Curiosity about AI quality

The session is live and remote, so a solid connection matters. Everything runs in-browser, and the no-code evaluation platform is provided.

Stable internet

AI engineers and product managers building, evaluating, or preparing to release AI features

Founders turning an AI prototype into a reliable product

Engineering and product leaders accountable for AI quality, safety, and release decisions

Agentic AI

Coding with AI

AI Workflows

Claude Code

OpenClaw

Vibe Coding

AI Evals

AI Transformation

RAG & Search

AI for PMs

AI for Engineers

AI for Designers

AI for Marketers

AI for Founders

Product Sense

Product Discovery

User Research

Prototyping

Growth

Analytics

Tech Foundations

Strategy

Influence

Leadership

Career Growth

Machine Learning

LLM Ops

Context Eng

Security

System Design

Figma

Design Systems

Visual Design

Design Strategy

Automation

Content Marketing

Demand Gen

Go-to-Market

Product Marketing

Positioning

Social Media

Brand

B2B Marketing

SEO & AEO

AI for Leaders

AI Governance

Communication

Management

People Operations

Exec Presence

Storytelling

Goal-setting

Personal Brand

Fundraising

Everyone

Operators

Data Scientists

Business Analysts

User Researchers

Customer Success

Project Managers

HR Professionals

Sales People

Lawyers

Finance

Investors

Real Estate

Educators

Creators

Introduction: Stop Shipping AI Agents on Vibes

The Challenge of Trusting AI Agents at Scale

The Gap Between a Demo and a Trusted Product

Why Traditional Software Testing Fails for AI Agents

Defining "Working": Perspectives from PM, Legal, and Ops

What is an Eval? Input, Expected Output, and Judge

A Four-Part Framework for Production AI Evals

The "New Hire" Framework for Scoping AI Agents

Orchestration Challenges and the Need for Circuit Breakers

Introducing "Probing": A Practical Framework for Evals

Conclusion and Call to Action

Product managers who drive strategy, development, and launch of products.

Engineers and engineering leaders who build, deploy, and scale software systems.

Applying AI tools and techniques to professional workflows — assign alongside more specific AI topics.

Building autonomous AI agent systems that take actions, use tools, and orchestrate multi-step workflows.

Measuring and evaluating AI model performance, quality, and reliability for production use cases.

Operating and deploying large language models in production — fine-tuning, inference, monitoring, and cost optimization.

Stop Shipping AI Agents on Vibes

What you'll learn

Why this topic matters

You'll learn from

KD Deshpande