AI Reliability: Evals and Harnesses for AI Agents You Can Trust

Tejas Kumar

AI Engineer at IBM, O'Reilly author

Measure whether your AI agent works, then make it work

Your agent works in the demo and then quietly fails for real users, and nobody on the team can say how often. I build agents at IBM and teach teams how to make them hold up, and what holds up is evals that tell you the truth plus a harness built around what they find.

In part 1 we write evals. We read real traces, name the failures, and turn them into code checks and an LLM judge you can trust because you've checked it against your own labels.

In part 2 we build a harness from an empty loop: tools, context management, guardrails, retries and a verify step, so the agent fails honestly instead of claiming success. The evals from part 1 tell us whether each change helped.

In part 3 we build live on your agents. Bring your code, your traces and your hardest questions, and we fix them together on screen.

You leave with evals and a harness for your own agents, on whatever models you use.

What you’ll learn

Ship AI agents you can prove work: evals that find the real failures and a harness that fixes them.

  • Read real traces and name each failure before writing a single eval

  • Write code checks and an LLM judge, then check the judge against your own labels

  • Go from an empty loop to tools, context management, guardrails and retries that hold on any model

  • Add a verify step so the agent fails honestly instead of claiming success

  • Move secrets and fragile steps out of the model and into code

  • Use your evals to prove each harness change helped, with numbers you can show your team

  • Bring your code to the live session and we fix your agent together on screen

Learn directly from Tejas

Tejas Kumar

Tejas Kumar

AI Engineer at IBM, best selling O'Reilly author, 80 recorded talks

See all products from Tejas

Who this course is for

  • Engineers shipping AI agents or LLM features to real users who need to know how often they fail

  • Tech leads deciding whether an agent is ready to ship, and how to prove it to their team

What's included

Tejas Kumar

Live sessions

Learn directly from Tejas Kumar in a real-time, interactive format.

Lifetime access

Go back to course content and recordings whenever you need to.

Community of peers

Stay accountable and share insights with like-minded professionals.

Certificate of completion

Share your new skills with your employer or on LinkedIn.

Maven Guarantee

Your purchase is backed by the Maven Guarantee.

Course syllabus

Week 1

Jan 13—Jan 17

    Jan

    13

    Part 1, Evals: measure what your agent actually does

    Wed 1/1311:00 AM—1:00 PM (EST)

Week 2

Jan 18—Jan 24

    Jan

    20

    Part 2, Harness: make your agent do the job

    Wed 1/2011:00 AM—1:00 PM (EST)

Frequently asked questions

Maven for Teams

Reimbursement

Get your company to pay

Everything L&D needs: email template, receipts, and certificate of completion.

Get reimbursed

Team discount

Learn with your teammates

Save 20%+ when 2 or more teammates enroll in the same cohort.

Save 20%+ with a team

Private cohort

Run a cohort for your org

A dedicated cohort with a custom schedule and curriculum, tailored to your team.

Book a private cohort

$1,999

USD

Jan 13—Jan 27
Enroll