WEBVTT

NOTE Nobody Reports the Misses: Counting False Passes in AI for Regulated Work
NOTE Generated by scripts/audio/build.mjs; do not edit by hand.

1
00:00:00.150 --> 00:00:05.255
Nobody Reports the Misses: Counting False Passes in AI for Regulated Work

2
00:00:05.855 --> 00:00:07.579
Wrong AI answers get caught.

3
00:00:07.859 --> 00:00:09.036
Missed items don't.

4
00:00:09.316 --> 00:00:12.809
How I'd count false passes in six kinds of regulated work.

5
00:00:13.409 --> 00:00:17.431
When an AI system gives a wrong answer, someone usually notices.

6
00:00:17.711 --> 00:00:20.085
The analyst reads it, frowns, fixes it.

7
00:00:20.365 --> 00:00:23.709
That error gets logged and shows up in the next quality review.

8
00:00:24.309 --> 00:00:29.848
When the system leaves something out, nobody notices, because there is nothing on the screen to frown at.

9
00:00:30.128 --> 00:00:32.461
The beneficial owner who was never listed.

10
00:00:32.741 --> 00:00:34.425
The exclusion nobody flagged.

11
00:00:34.705 --> 00:00:36.134
The clause the review skipped.

12
00:00:36.414 --> 00:00:44.455
The output looks complete, so it passes, and the cost arrives months later as a fine, a denied claim or a deal that falls over.

13
00:00:45.055 --> 00:00:47.862
Most accuracy numbers count the first kind of error.

14
00:00:48.142 --> 00:00:53.656
The work I care about is counting the second, and setting the bar for when a step can run without a human.

15
00:00:54.856 --> 00:00:55.674
What I measured

16
00:00:56.274 --> 00:00:58.767
I learned this from harnesses I built myself.

17
00:00:59.047 --> 00:01:13.724
CrossSource is a public one that checks whether a legal research answer's citations actually support its claims, with one model (claude-opus-4-8) generating and judging under two prompt configurations, baseline and strict.

18
00:01:14.004 --> 00:01:17.142
I expected the main failure to be wrong citations.

19
00:01:17.422 --> 00:01:20.963
It wasn't; the citations the answers gave mostly held up.

20
00:01:21.243 --> 00:01:23.075
The dominant failure was omission.

21
00:01:23.355 --> 00:01:30.390
The authority a lawyer would have needed often never made it into the retrieved set, so no amount of model quality could have cited it.

22
00:01:30.670 --> 00:01:35.194
What that shows: a precision-style score flatters a system that misses things.

23
00:01:35.474 --> 00:01:39.581
What it doesn't show: how often this happens in any given production product.

24
00:01:39.861 --> 00:01:42.205
It is one harness, on public case law.

25
00:01:42.805 --> 00:01:44.603
The second number is about judges.

26
00:01:44.883 --> 00:01:53.178
In a separate audit of AI search answers, I labelled 40 answers blind, then had two LLM judges tag the same answers.

27
00:01:53.458 --> 00:01:58.685
Exact agreement with my labels: 8% for one judge, 30% for the other.

28
00:01:58.965 --> 00:02:01.410
That doesn't make LLM judges useless.

29
00:02:01.690 --> 00:02:07.497
A good part of the gap was definitional; the judges and I read some failure categories differently.

30
00:02:07.777 --> 00:02:15.627
It does mean a judge's agreement with humans is a property of the task, and you measure it before you trust a single number the judge produces.

31
00:02:16.227 --> 00:02:25.745
The third: two cheap-tier screening models, claude-haiku-4-5 and gpt-5-mini, the same 21 screening cases, one verdict each.

32
00:02:26.025 --> 00:02:30.132
The verdict flipped on 15 of 21 depending on which model ran it.

33
00:02:30.412 --> 00:02:32.410
That doesn't tell you which model was right.

34
00:02:32.690 --> 00:02:39.878
It tells you the choice of model can move outcomes more than the inputs you think you're testing, and a single-model pass rate hides that.

35
00:02:40.478 --> 00:02:46.195
The errors that cost money are the ones nobody sees, and the tools that grade them need grading too.

36
00:02:46.475 --> 00:02:49.643
Here is what that looks like in six kinds of regulated work.

37
00:02:50.843 --> 00:02:54.043
Compliance and KYB: the missed beneficial owner

38
00:02:54.643 --> 00:02:59.740
An onboarding agent reads the registry filings and returns a clean ownership chart.

39
00:03:00.020 --> 00:03:08.480
It is wrong by omission: a significant holder sits two layers up, behind a holding company in another jurisdiction, and never appears.

40
00:03:08.760 --> 00:03:11.419
Nobody flags a person who isn't on the page.

41
00:03:11.699 --> 00:03:18.244
The customer pays when a regulator or a partner bank finds that owner later and asks why the file passed.

42
00:03:18.524 --> 00:03:27.876
To count it: take a sample of entities, have a compliance analyst build the full ownership tree from source documents, and measure owner-level recall against it.

43
00:03:28.156 --> 00:03:31.995
A file passes only if every owner above the threshold is found.

44
00:03:33.195 --> 00:03:36.047
Audit and controls: the false pass on a control

45
00:03:36.647 --> 00:03:42.300
An agent tests a control, say approvals on payments above a threshold, and marks it effective.

46
00:03:42.580 --> 00:03:48.266
The sample it pulled missed the exceptions, or it accepted an approval dated after the payment went out.

47
00:03:48.546 --> 00:03:49.910
The workpaper says pass.

48
00:03:50.190 --> 00:03:56.440
The customer pays at the next external audit, when a deficiency surfaces that the tool had already signed off.

49
00:03:56.720 --> 00:04:05.268
To count it: seed a test population with known exceptions, have an experienced auditor label which items fail, and measure exception recall.

50
00:04:05.548 --> 00:04:11.617
Any run that calls a seeded failure "effective" is a false pass, and one false pass fails the test.

51
00:04:12.817 --> 00:04:16.099
Insurance: the coverage gap the proposal never flagged

52
00:04:16.699 --> 00:04:22.604
A broker's AI compares a client's exposures to a proposed policy and writes a clean summary.

53
00:04:22.884 --> 00:04:31.125
It never mentions that flood is excluded at the client's main warehouse, or that the cyber sublimit sits below the cost of their last incident.

54
00:04:31.405 --> 00:04:32.441
The client signs.

55
00:04:32.721 --> 00:04:37.774
The gap shows up as a denied claim, and the broker's errors-and-omissions carrier hears about it.

56
00:04:38.054 --> 00:04:47.157
To count it: take past proposals where a senior broker or claims reviewer has listed the real gaps, run the system on them, and measure gap recall.

57
00:04:47.437 --> 00:04:53.519
A proposal passes only if every material exclusion or sublimit the reviewer found is flagged.

58
00:04:54.719 --> 00:04:57.055
Legal: the clause the review didn't surface

59
00:04:57.655 --> 00:05:03.669
A contract review tool returns a tidy issues list: indemnity, liability cap, governing law.

60
00:05:03.949 --> 00:05:12.028
What it didn't surface is the auto-renewal with a 90-day notice window, or the assignment clause that blocks the client's planned acquisition.

61
00:05:12.308 --> 00:05:14.888
The lawyer trusted the list because it looked complete.

62
00:05:15.168 --> 00:05:18.355
The client pays when the window closes or the deal stalls.

63
00:05:18.635 --> 00:05:26.664
To count it: have a lawyer mark up a sample of contracts against the client's playbook, issue by issue, and measure recall per clause type.

64
00:05:26.944 --> 00:05:31.153
A review passes when it surfaces every playbook deviation the lawyer marked.

65
00:05:31.433 --> 00:05:33.800
Extra comments don't buy back a missed one.

66
00:05:35.000 --> 00:05:37.414
Patents: the prior art the search never found

67
00:05:38.014 --> 00:05:40.987
A prior art search returns twenty strong references.

68
00:05:41.267 --> 00:05:47.812
The one that matters, an old filing in another language or a conference paper using different terms, isn't among them.

69
00:05:48.092 --> 00:05:56.154
The customer pays when that reference turns up in an opposition or invalidity challenge, after the filing and prosecution money is spent.

70
00:05:56.434 --> 00:06:05.861
To count it: use cases where the decisive art is already known from examiner citations or past challenges, hide it, and check whether the search finds it.

71
00:06:06.141 --> 00:06:08.707
The record, or a patent professional, is the label.

72
00:06:08.987 --> 00:06:14.234
A search passes only if the decisive reference lands in what a reviewer actually reads.

73
00:06:15.434 --> 00:06:18.837
Regulated voice agents: the disclosure the agent skipped

74
00:06:19.437 --> 00:06:23.157
A voice agent handles a collections or insurance call well.

75
00:06:23.437 --> 00:06:25.203
Tone and payment plan are right.

76
00:06:25.483 --> 00:06:30.969
It never said the call was recorded, or it dropped a required disclosure when the caller interrupted.

77
00:06:31.249 --> 00:06:33.433
Nobody listens to the calls that went well.

78
00:06:33.713 --> 00:06:37.111
The customer carries the regulatory exposure, call by call.

79
00:06:37.391 --> 00:06:43.837
Healthcare admin has the same shape: the prior-auth request filed without the one document the payer needed.

80
00:06:44.117 --> 00:06:51.795
To count it: have a compliance reviewer label a call sample for each required disclosure, and measure disclosure recall per call.

81
00:06:52.075 --> 00:06:56.526
A call passes only if every required item is present, at the right point.

82
00:06:57.726 --> 00:06:58.888
How I'd set the bar

83
00:06:59.488 --> 00:07:02.984
Ground truth per failure type, not one gold answer per case.

84
00:07:03.264 --> 00:07:12.135
A missed owner, a missed exception and a missed disclosure need different labellers, and the labeller has to be someone the customer would trust to sign the file.

85
00:07:12.735 --> 00:07:15.534
An acceptance bar written as a number before launch.

86
00:07:15.814 --> 00:07:23.600
"Owner recall at or above an agreed figure on the labelled sample, with zero misses above the reporting threshold" is a bar.

87
00:07:23.880 --> 00:07:25.433
"High accuracy" is not.

88
00:07:25.713 --> 00:07:29.798
Writing it first stops the bar drifting toward whatever the system scores.

89
00:07:30.398 --> 00:07:34.993
A short list of the failure modes that matter, ranked by what they cost the customer.

90
00:07:35.273 --> 00:07:38.401
A misspelled name and a missing owner shouldn't share a metric.

91
00:07:39.001 --> 00:07:40.322
A rule for autonomy.

92
00:07:40.602 --> 00:07:51.678
A step runs without a human only when its miss rate on the labelled sample holds under the bar across releases, and any judge used to score it has its own measured agreement with humans.

93
00:07:51.958 --> 00:07:56.286
Until then a human reviews it, and every review is kept as new labels.

94
00:07:57.486 --> 00:07:58.389
Compare notes

95
00:07:58.989 --> 00:08:08.849
If you're measuring misses in production, in any of these domains or one I haven't listed, I'd like to compare notes: what you label, who labels it, and where you set the bar.

96
00:08:09.129 --> 00:08:11.182
Most dashboards count what the system said.

97
00:08:11.462 --> 00:08:13.351
I'm more interested in what it didn't.