Agent Evaluation and Reliability Kit

You’ll learn how to measure the six signals that matter most for production agents: task completion, tool-call accuracy, latency, cost per run, retry rate, and escalation rate.

You’ll also learn how to:

• Build and maintain a golden test set

• Cover happy-path, edge, adversarial, and regression cases

• Grade deterministically where possible

• Use and calibrate LLM judges

• Track prompt and test-set versions

• Turn every production bug into a regression test

• Run repeatable checks before releasing agent changes