mirror-eval · judge check · 40 labels

Two judges, 40 blind labels. Matches: 3 and 12.

3 of 40 and 12 of 40 matched.

The note

Fig. 1 Match: whole tag set equals mine.

Two LLM judges against 40 answers I had labelled blind: the Claude judge matched my tag set exactly on 3 of 40, the Gemini judge on 12 of 40.

The 40 came from my AI search audit, labelled by hand before I opened either judge's tags. One answer I had tagged clean got "unverifiable_specific" from one judge and "fabricated_metric" from the other.

The sample was built to be hard: 16 answers where the two judges had disagreed with each other, 12 high-impact ones, 12 clean controls. A match meant the whole tag set, not one overlapping tag. So 3 of 40 is a hard-case rate under a strict rule, not the judges' rate on everything.

It was still far below what I needed to report a single number per category. What I shipped instead is a band per category: the floor is answers where both judges fired the tag, the ceiling is answers where either did. A band up to 10 points wide with enough concordance between the judges gets called narrow, up to 20 directional, wider than that uninformative; in the second wave none qualified as narrow. Two categories only one judge ever fired on were barred from the headline numbers.

Limits. Judges were claude-haiku-4-5-20251001 and gemini-3.8-flash. One labeller, me; the labelling notes are withheld and available on request.

If no category's band ever qualifies as narrow, is the band still a result, or a polite way of saying the judge cannot be used yet?

Where the numbers live

One labeller. A hard-case rate, not the overall rate. Judges claude-haiku-4-5-20251001 and gemini-3.8-flash.