Before Advanced AI Evals: Free Readings, and Where the Basics Stop at High Stakes

Ten free useful readings by the people who teach the basics, in a set order, with a note on what each one leaves open when the stakes are high. Start with one question: is a wrong verdict from your eval cheap or costly? If it is cheap, the basics are enough. If it is costly, you need to know how far to trust each judge, at each level of task difficulty, and say so with a number.

1. Decide your stakes first. A one-page table shows when one validated judge and a pass rate are enough, and when they are not: clinical, credit, hiring, legal, safety, or anything a regulator, customer or board will audit.

2. What good basic courses already teach. Error analysis, one judge per failure mode, 100 to 200 labels per failure mode, judge-corrected pass rates with intervals, and criteria drift. We credit it and build on it.

3. Where the basics are wrong at high stakes, with sources. Four widely taught positions (banning rating scales, one correction per judge whatever the task level, accepting scores that cannot be compared over time, same-model judges as "usually fine"), each with sources and our labeled evidence.

4. A gap map. Twelve gaps, from judge harshness that shifts with task difficulty to differential item functioning (DIF), shared errors, dimensionality and metrological traceability, each tagged with what goes wrong at high stakes and the kind of evidence behind it.

5. Evidence you can check. Every claim is labeled REAL, SIMULATION, ILLUSTRATION or PUBLISHED DATA, with limits stated.

6. Ten core readings plus a curated library. Seven required (about 5.5 hours), all ten about 8 hours, an optional 2-hour on-ramp for people new to evals, and further free videos, posts, papers, books and tools. Each entry says what you'll learn and its limit at high stakes.

7. A readiness check. Four things you should be able to do before an advanced class, in your own words.

• “Our LLM judge says 82%, and we cannot tell if that is real.”

• “We re-ran the judge and got a different result.”

• “We do not know which of the many eval articles to read first.”

• “Our logs show what ran, but not whether the score means anything.”

• “We cannot say how sure we are when we report a score.”

A judge’s verdict often decides what ships, what is fixed and what is trusted. When the verdict is wrong, the cost can be a missed defect, a wasted budget or a late decision. Most teams can still fix this: a few ideas, read in the right order, change what you ask of your own eval. The guide helps you spend care where a wrong verdict costs the most, and keep it simple where it does not.

Engineers, product leads, data scientists and quality leaders who use an LLM to judge AI output, or who will be asked to defend that judge’s verdict. New to evals? The guide includes an optional free on-ramp of about 2 hours.

• Before the 1-hour Lightning Lesson: recommended. About 20 minutes. The readings are optional for the lesson.

• Before courses and workshops: required. Read the seven marked required readings (about 5.5 hours) and pass the readiness check. All ten readings take about 8 hours.

• Bring one judge you use today, with a few dozen of its ratings, and the one failure that would hurt your users most.

Good basic courses already correct a judge's pass rate for its errors and put a confidence interval on it. What one corrected pass rate cannot show is a judge whose harshness changes with how complex the task is. In our certified analysis of one record fit (REAL, descriptive; no credential of record from this fit yet), across Model of Hierarchical Complexity (MHC) levels 7 to 11, the most lenient and the most severe judge sit 2.6 to 3.6 logits apart, and some judges drift by more than a logit between neighbouring levels (up to 1.7 logits between L7 and L11), so one severity correction per model is not enough. This rests on 76,617 judge reads of 5,346 texts. One correction averages over levels; calibrating by level shows whether that matters for your eval. The guide shows how.