Free Lesson
Build AI Agent Evals from Production Traces
60 min
Dec 3, 2026 12:00 PM
By continuing, you agree to Maven's Terms and Privacy Policy.
What you'll learn
Turn real agent runs into a test set
Pick traces from real use and label what a good answer looks like.
Score answers with checks and an LLM judge
Code checks for facts and format, a judge for the rest, tuned to your labels.
Fail the build when the agent gets worse
Run the evals in CI on every prompt or model change.
Why this topic matters
One prompt or model change can quietly break an agent that worked yesterday. Most teams test by trying a few questions by hand, so their users find the regressions first. Live, I turn real agent runs from Langfuse into a test set, score them with code checks and an LLM judge, and fail the build when a change makes the agent worse. Part 6 of 7 in the Agentic AI Systems series.
