nobody reports the misses · six cases
Wrong answers get caught. Missed items do not.
The note
Retrieval missed the answer passage on 6 of the 25 questions in my legal citation eval. On five of those six the model wrote around the gap, and the answer read fine.
On the sixth the model said it could not answer. Retrieval missing the passage was the most common failure in the run.
Precision looked clean, because precision scores only what is on the page. The miss showed up only against a list of what should have been there.
I think this is the under-reported error in AI for regulated work. A wrong line gets caught in review. A missing one waits until someone else finds it. So in an essay I took six kinds of work and asked the same four questions of each: what does the miss look like, who pays, how would you count it, who would label it.
KYB: an owner two layers up who is never on the page. SOX: a control marked effective after a test sample that missed the exceptions. Claims: an exclusion left unflagged. Contracts: a renewal clause the summary skipped. Patents: the decisive reference that was not found. Voice: a disclosure the agent did not say.
As I read them, all six counts come out as recall tests, run against a list of what should have been there.
Limits. The six cases are hypothetical, written to show how I would count, with no incident and no cost figure. The 6 of 25 is one model on 25 questions over public-domain case law, in areas such as contracts, torts and employment, nothing wider.
Who on your team keeps the list of what should have been there, and does anyone score against it?
Where the numbers live
The essay, with the six cases and the table
Six hypothetical cases, no incident or cost figure. Behind it: one model, 25 questions on public-domain case law.