Same default model writes and judges. 15 labels.

Two sheets swing in and face each other like a mirror: GENERATOR and JUDGE, both claude-opus-4-8, the default for each in my citation eval.

Fifteen of the judge's verdicts were labelled blind by a human who is not a lawyer, and 15 of 15 agreed. The exact 95 percent interval for 15 of 15 starts at 0.78, so the honest reading is "probably above 78 percent", not "perfect".

The eval's limitations section names the problem: judge and generator share a model family, a known self-preference bias, and the human agreement check is the mitigation, not a cure.

The check took every citation the judge called unsupported, six, plus a seeded random draw of ones it called supported, nine. The sample came from the run before the span-splitter bug was fixed, so three of the six unsupported verdicts were the comma spans from the judge post, and the labeller agreed those were unsupported too.

What it does not reach: it covers citation precision verdicts only. The faithfulness score of 1.000 and the relevance scores come from the same judge and were not checked by a human at all. The sample was drawn by verdict, not by question or by prompt configuration. There is no second judge model family; the README lists that as not built. There is no second rater either.

It tells me the judge was not inventing unsupported citations in the sample. It does not tell me what a different model family would have said about the same spans.

Which would you pay for first: a second rater on these 15 verdicts, or a second judge family on all of them?

Where the numbers live

• README.md
• src/judge_check.py
• src/generate.py
• src/evaluate.py
• results/judge_agreement.md

The eval, with the judge check script and the limitations section

One non-lawyer rater. Three of the six unsupported were the comma bug. No second judge family.