Nobody Reports the Misses: Counting False Passes in AI for Regulated Work

I learned this from harnesses I built myself. CrossSource is a public one that checks whether a legal research answer's citations actually support its claims, with one model (claude-opus-4-8) generating and judging under two prompt configurations, baseline and strict. I expected the main failure to be wrong citations. It wasn't; the citations the answers gave mostly held up. The dominant failure was omission. The authority a lawyer would have needed often never made it into the retrieved set, so no amount of model quality could have cited it. What that shows: a precision-style score flatters a system that misses things. What it doesn't show: how often this happens in any given production product. It is one harness, on public case law.

The second number is about judges. In a separate audit of AI search answers, I labelled 40 answers blind, then had two LLM judges tag the same answers. Exact agreement with my labels: 8% for one judge, 30% for the other. That doesn't make LLM judges useless. A good part of the gap was definitional; the judges and I read some failure categories differently. It does mean a judge's agreement with humans is a property of the task, and you measure it before you trust a single number the judge produces.

The third: two cheap-tier screening models, claude-haiku-4-5 and gpt-5-mini, the same 21 screening cases, one verdict each. The verdict flipped on 15 of 21 depending on which model ran it. That doesn't tell you which model was right. It tells you the choice of model can move outcomes more than the inputs you think you're testing, and a single-model pass rate hides that.

The errors that cost money are the ones nobody sees, and the tools that grade them need grading too. Here is what that looks like in six kinds of regulated work.

An onboarding agent reads the registry filings and returns a clean ownership chart. It is wrong by omission: a significant holder sits two layers up, behind a holding company in another jurisdiction, and never appears. Nobody flags a person who isn't on the page. The customer pays when a regulator or a partner bank finds that owner later and asks why the file passed. To count it: take a sample of entities, have a compliance analyst build the full ownership tree from source documents, and measure owner-level recall against it. A file passes only if every owner above the threshold is found.

An agent tests a control, say approvals on payments above a threshold, and marks it effective. The sample it pulled missed the exceptions, or it accepted an approval dated after the payment went out. The workpaper says pass. The customer pays at the next external audit, when a deficiency surfaces that the tool had already signed off. To count it: seed a test population with known exceptions, have an experienced auditor label which items fail, and measure exception recall. Any run that calls a seeded failure "effective" is a false pass, and one false pass fails the test.

A broker's AI compares a client's exposures to a proposed policy and writes a clean summary. It never mentions that flood is excluded at the client's main warehouse, or that the cyber sublimit sits below the cost of their last incident. The client signs. The gap shows up as a denied claim, and the broker's errors-and-omissions carrier hears about it. To count it: take past proposals where a senior broker or claims reviewer has listed the real gaps, run the system on them, and measure gap recall. A proposal passes only if every material exclusion or sublimit the reviewer found is flagged.

A contract review tool returns a tidy issues list: indemnity, liability cap, governing law. What it didn't surface is the auto-renewal with a 90-day notice window, or the assignment clause that blocks the client's planned acquisition. The lawyer trusted the list because it looked complete. The client pays when the window closes or the deal stalls. To count it: have a lawyer mark up a sample of contracts against the client's playbook, issue by issue, and measure recall per clause type. A review passes when it surfaces every playbook deviation the lawyer marked. Extra comments don't buy back a missed one.

A prior art search returns twenty strong references. The one that matters, an old filing in another language or a conference paper using different terms, isn't among them. The customer pays when that reference turns up in an opposition or invalidity challenge, after the filing and prosecution money is spent. To count it: use cases where the decisive art is already known from examiner citations or past challenges, hide it, and check whether the search finds it. The record, or a patent professional, is the label. A search passes only if the decisive reference lands in what a reviewer actually reads.

A voice agent handles a collections or insurance call well. Tone and payment plan are right. It never said the call was recorded, or it dropped a required disclosure when the caller interrupted. Nobody listens to the calls that went well. The customer carries the regulatory exposure, call by call. Healthcare admin has the same shape: the prior-auth request filed without the one document the payer needed. To count it: have a compliance reviewer label a call sample for each required disclosure, and measure disclosure recall per call. A call passes only if every required item is present, at the right point.

Ground truth per failure type, not one gold answer per case. A missed owner, a missed exception and a missed disclosure need different labellers, and the labeller has to be someone the customer would trust to sign the file.

An acceptance bar written as a number before launch. "Owner recall at or above an agreed figure on the labelled sample, with zero misses above the reporting threshold" is a bar. "High accuracy" is not. Writing it first stops the bar drifting toward whatever the system scores.

A short list of the failure modes that matter, ranked by what they cost the customer. A misspelled name and a missing owner shouldn't share a metric.

A rule for autonomy. A step runs without a human only when its miss rate on the labelled sample holds under the bar across releases, and any judge used to score it has its own measured agreement with humans. Until then a human reviews it, and every review is kept as new labels.

If you're measuring misses in production, in any of these domains or one I haven't listed, I'd like to compare notes: what you label, who labels it, and where you set the bar. Most dashboards count what the system said. I'm more interested in what it didn't.