decisions api · hearsay · two prompts
The benchmark's one-liner: 1 of 9. Four sentences: 3 of 8.
The note
Two prompt cards and ten item cards. The one-line card runs the ten: 9 answered, 1 right. The deck resets, and the four-sentence card runs them again: 8 answered, 3 right.
The one-line card is LegalBench's hearsay instruction: "Hearsay is an out-of-court statement introduced to prove the truth of the matter asserted. Is there hearsay?" Through OpenAI's Decisions API, on that ten-item pilot, it got 1 right of the 9 it answered. One item it refused.
I rewrote the instruction as four sentences of the rule: the definition, that a statement made in court during the current trial is not hearsay, that non-verbal conduct counts only if intended as an assertion, and that a statement offered for a purpose other than its truth is not hearsay. On the same ten items it got 3 of the 8 it answered, and refused two. On the full 94 hearsay items it scored 74 percent on the 73 it answered.
That headline number holds on its own terms. Underneath, the benchmark's own one-liner is not what it was measured against. Anyone comparing my 74 percent to a LegalBench leaderboard should know the instruction differs. One item answered under the short prompt was refused under the long one. In the full run hearsay was the worst of the six tasks, and the only one that drew refusals at all.
Limits. The pilot is ten rows, so 1 of 9 and 3 of 8 are not rates, they are a reason to look closer. One run, one model, gpt-6-luna. The labels are the benchmark's.
Should a score earned under your rewritten instruction ever sit in the same table as scores earned under the benchmark's own?
Where the numbers live
The quiz, both hearsay prompts and the pilot file
Ten rows is a reason to look, not a rate. One run, gpt-6-luna, 8 Oct 2026. Labels are LegalBench's.