At 0.9 confidence, 47 of 377 were wrong.

Is there a confidence score above which a legal answer can skip the check? On 544 yes or no legal questions, OpenAI's Decisions API answered 523 and was 0.9 or more sure on 377. 47 of those 377 were wrong.

I sent each question through the API once and kept the probability it returned with every answer.

At 0.9 it looks usable. 330 of the 377 matched the benchmark labels: accuracy 87.5 percent on 72 percent of the work.

Underneath, raising the bar barely moved it. At 0.95: 87.8 percent on 64 percent. At 0.99: 90.3 percent on 49 percent. Even the 224 answers where it reported a probability of exactly 0 or 1 came in at 92 percent. On this data there was no threshold that bought 95 percent.

The second row of my results file had said so before any curve did: a contract clause labelled yes, and the model said no at a confidence of 0.99.

Limits. 50 to 100 items per task, so each task's accuracy carries a margin of roughly 7 to 10 points. The labels are LegalBench's and I did not audit the ones the model got wrong. The thresholds were read off the same items they were scored on. Model gpt-6-luna, one run, 8 October.

If the most certain bucket on this data is 92 percent right, which of your work would you still let through unchecked?

Where the numbers live

• README.md
• judge.py
• results/items.csv
• results/metrics.json

The quiz, the trust curve and the per-task table

50 to 100 items per task, a margin of 7 to 10 points each. Labels not audited. gpt-6-luna, 8 Oct 2026.