224 answers at exactly 0 or 1. Eighteen were wrong.

A ruler with two end stops, 0 and 1. Tiles stack at each stop, three answers to a tile, and six of them flip red.

Those are the answers in my legal quiz that came back at a probability of exactly 0 or exactly 1: 224 of them. Eighteen were wrong by the benchmark's labels; 206 were right. Ninety-two percent.

The quiz was 544 yes or no legal questions through OpenAI's Decisions API, one call each, scored against LegalBench's labels. No threshold between 0.5 and 0.99 had reached 95 percent accuracy. The round numbers were the last place to look: if a reported 0 or 1 meant anything stronger than 0.99, the 224 should have cleared it. They came in 1.7 points above the 0.99 band, and three points short of 95.

So the probability field tops out before the accuracy does. A pipeline that treats a probability of 0 or 1 as no review needed would wave through 18 of these 224. One of them is a change-of-control clause labelled present, returned at a probability of exactly 0. Most of the 224 sit at 0 rather than 1: 166 of them, and 17 of the 18 misses.

A reported 0 or 1 is a bin, not a promise, and the bin has an error rate you can measure in an afternoon.

Limits. One run, one model, gpt-6-luna. The labels are the benchmark's and I did not audit the 18. Fifty to a hundred items per task, so each task's rate carries a margin of roughly 7 to 10 points. The 224 come from five of the six tasks; the jurisdiction task returned none, and 12 of the 18 misses are change-of-control clauses.

Should a reported 0 ever skip review, when 17 of the 18 misses at the end stops came back as exactly 0?

Where the numbers live

• items.csv
• metrics.json
• judge.py
• trust_curve.csv
• FINDINGS.md

The quiz and the trust curve; the per-item file in the repo holds the 224

One run, gpt-6-luna, 8 Oct 2026. Labels are LegalBench's, misses not audited. Five of the six tasks.