I set the floor at 60%. Agreement came in at 35.

A line drawn at 60 percent across a field. Columns rise against it: ChatGPT to 73, Gemini 33, Claude 22, Perplexity 13. The last, All, stops at 35.

That is the second wave of my AI search audit. The two judges produced the same tag set on 117 of 332 answers, 35 percent, against a floor of 60 I had written into the predictions file in August.

Below 60, the file says, means the taxonomy is underspecified, not that the engines are erratic. The judges were claude-haiku-4-5-20251001 and gemini-3.8-flash, measured on 4 September. The Claude judge averaged 1.62 tags per answer; the Gemini judge 0.55.

The number did not arrive cold. A first pass on the first wave had agreed 8 percent of the time. A tightening, two categories moved out of the judges' hands into code, precedence rules, worked examples, took it to 17. The second wave under the shared config: 35. The first wave re-judged under that same config: 18.

Still under the line, and the file had already said what that meant. What the repo does with the verdict: nine of eleven categories are reported as bands, with a floor where both judges fired the tag and a ceiling where either did, instead of as rates. Two categories only one judge ever fired on are barred from the headline.

Limits. 332 answers in the scored wave, both waves judged under the shared config, two judge families, and the blind check against my own labels failed for both judges.

Is 35 percent a verdict on the taxonomy, or on pairing a judge that averages 1.62 tags per answer with one that averages 0.55?

Where the numbers live

• README.md
• PREDICTIONS.md
• Site page /notes/taxonomy-miss/
• scores.jsonl
• taxonomy.yaml

One wave, two judge families, and both judges had failed the blind check.