The judge caught a bug I didn't
On validating LLM-as-judge evals — and why a 15/15 blind agreement and a broken instrument can live in the same pipeline.
My eval judge agreed with me 15 times out of 15. That's not the interesting part.
For CrossSource — an open harness I built to measure citation accuracy in legal RAG — I use an LLM judge to score whether each cited passage actually supports the claim. Standard practice. Also the part of the pipeline everyone trusts by default.
I didn't want to trust it by default. So I hand-graded a blind, stratified sample of its verdicts: 15/15 agreement. Validation passed.
The judge didn't catch the bug. The act of checking the judge caught the bug.
But the validation pass itself surfaced something the judge and I had both been getting wrong: consecutive citations were producing punctuation-only claim spans — fragments like "." — which the judge was dutifully scoring as citation failures. A harness bug, not a model failure. Fixing it changed the metrics: citation precision moved from 0.981 to 0.994 under strict citation discipline.
That's the lesson I keep re-learning: an unvalidated eval reports wrong numbers with full confidence — which is precisely the failure mode the eval exists to catch. A 15/15 validation and a broken instrument can coexist in the same pipeline. You only find out which world you're in by grading blind against ground truth.
The harness, data, and method are public — including the failure taxonomy and the validation protocol.
Read the CrossSource case study · Read the code
If you own eval quality for an LLM product: what's your validation protocol for the judge itself?
If you're shipping an LLM product, let's talk about what your evals miss.