Eval methodology · CrossSource

The judge caught a bug I didn't

On validating LLM-as-judge evals — and why a 15/15 blind agreement and a broken instrument can live in the same pipeline.

· 2 min read

What I checked

My eval judge agreed with me 15 times out of 15. That's not the interesting part.

For CrossSource — an open harness I built to measure citation accuracy in legal RAG — I use an LLM judge to score whether each cited passage actually supports the claim. Standard practice. Also the part of the pipeline everyone trusts by default.

I didn't want to trust it by default. So I hand-graded a blind, stratified sample of its verdicts: 15/15 agreement. Validation passed.

The judge didn't catch the bug. The act of checking the judge caught the bug.

The bug the validation surfaced

But the validation pass itself surfaced something the judge and I had both been getting wrong: consecutive citations were producing punctuation-only claim spans — fragments like "." — which the judge was dutifully scoring as citation failures. A harness bug, not a model failure. Fixing it changed the failure counts: unsupported_claim went from 3 to 0, because all three were this artifact, and the affected items were re-scored. The citation precision move from 0.981 to 0.994 is a different comparison, the baseline prompt against the strict citation-discipline prompt.

Fig. 1 One answer through the harness

Transcript

  1. The answer. The model answers a legal question and cites its sources inline.
  2. Claim spans. The harness cuts the answer into claim spans. The judge will check each one against the passage it cites.
  3. The bug. Where two citations sit side by side, a span is left holding nothing but punctuation, a fragment like "." that is not a claim.
  4. The judge scores it. The judge dutifully scores that span as a citation failure. A harness bug now looks like a model failure.
  5. The blind check. I hand-graded a blind, stratified sample of the judge's verdicts: 15 of 15 agreed. Checking the judge is what exposed the bug.

Source: this essay and the CrossSource case study. The spans drawn are placeholders, not real answer text. Case study →

Why this keeps happening

That's the lesson I keep re-learning: an unvalidated eval reports wrong numbers with full confidence — which is precisely the failure mode the eval exists to catch. A 15/15 validation and a broken instrument can coexist in the same pipeline. You only find out which world you're in by grading blind against ground truth.

More

The harness, data, and method are public — including the failure taxonomy and the validation protocol.

Read the CrossSource case study · Read the code

If you own eval quality for an LLM product: what's your validation protocol for the judge itself?

AI PRODUCT MANAGER · LLM EVALUATION & RAG QUALITY

If you're shipping an LLM product, let's talk about what your evals miss.

View the repo → Book a call