Free Lesson
Advanced AI Evaluation: Beyond Pass Rates
60 min
Jan 7, 2027 9:00 AM
By continuing, you agree to Maven's Terms and Privacy Policy.
What you'll learn
See what a pass rate really contains
Name the judge harshness (or leniency, item difficulty and noise that ride along with model quality in any score.
Correct for judge harshness, add error bars
Measure each judge's harshness at different levels, remove the bias, and report the result with its error bar.
Make a ship call you can defend
Decide whether a result clears your standard, and fix a stopping rule before you run.
Why this topic matters
Pass rates are easy to compute and hard to trust. Swap the LLM judge and identical answers can earn different verdicts, so a move from 79 to 82 may be the judge, the item set or noise. Hamel Husain and Shreya Shankar's AI Evals course is an excellent foundation. This class is for experts ready to go deeper: why evals mislead, and how to make them engineering-worthy.
