WEBVTT

NOTE I gave OpenAI's Decisions API a quiz written by lawyers
NOTE Generated by scripts/audio/build.mjs; do not edit by hand.

1
00:00:00.150 --> 00:00:04.231
I gave OpenAI's Decisions API a quiz written by lawyers

2
00:00:04.831 --> 00:00:16.994
544 yes/no legal questions, one API call each, and the question that matters to anyone routing work on a confidence score: is there a number above which you can stop checking?

3
00:00:17.274 --> 00:00:18.321
Not in this data.

4
00:00:18.921 --> 00:00:23.681
OpenAI's Decisions API is a model endpoint that does not write anything.

5
00:00:23.961 --> 00:00:28.056
You send it a piece of text and a question, and it sends back a probability.

6
00:00:28.336 --> 00:00:37.173
No prose, no output tokens, ten cents per million input tokens, and the docs say about ten times faster than the Responses API.

7
00:00:37.453 --> 00:00:40.108
It is built for routing, filtering and triage.

8
00:00:40.708 --> 00:00:41.907
The pitch is the number.

9
00:00:42.187 --> 00:00:48.227
A probability invites a threshold: above it the system handles the item, below it a person does.

10
00:00:48.507 --> 00:00:51.404
So the test I cared about was not "is it accurate".

11
00:00:51.684 --> 00:00:53.944
It was "when it says it is sure, is it right".

12
00:00:54.224 --> 00:01:00.409
Those are different questions, and only the second one decides whether the thing can take work off a reviewer's desk.

13
00:01:01.009 --> 00:01:05.588
So I gave it a quiz lawyers wrote, and watched what happened to the confident answers.

14
00:01:06.788 --> 00:01:07.427
The quiz

15
00:01:08.027 --> 00:01:18.069
LegalBench is a public benchmark from 2023, 162 tasks contributed by lawyers and legal academics, released under CC BY.

16
00:01:18.349 --> 00:01:20.493
I took six of its yes/no tasks.

17
00:01:20.773 --> 00:01:31.557
Three are contract clause classification from CUAD: does this clause restrict assignment, does it restrict competing, does it give one party rights if the other changes control.

18
00:01:31.837 --> 00:01:38.182
Two are short fact patterns that test legal reasoning: is there hearsay, is there personal jurisdiction.

19
00:01:38.462 --> 00:01:45.203
One is consumer contract Q&A: read an excerpt from a terms of service and answer a question about it.

20
00:01:45.803 --> 00:01:52.746
That gave 544 items. 100 per task, half yes and half no, where the task had enough rows.

21
00:01:53.026 --> 00:02:00.487
Every test item for the two reasoning tasks, which are small: 94 for hearsay and 50 for personal jurisdiction.

22
00:02:01.087 --> 00:02:02.293
One call per item.

23
00:02:02.573 --> 00:02:09.592
The clause or fact pattern as the input, the task's own instruction line as the question, one predicate question.

24
00:02:09.872 --> 00:02:13.142
No examples, no system prompt, no second attempt.

25
00:02:13.422 --> 00:02:20.764
The whole run was 192 thousand input tokens, pilot included, which cost two cents and took four minutes.

26
00:02:21.364 --> 00:02:23.680
Scoring is the simplest thing that could work.

27
00:02:23.960 --> 00:02:28.015
The model's answer is yes if the probability is at least one half.

28
00:02:28.295 --> 00:02:34.189
Its confidence is the distance from that fence: the probability for a yes, one minus it for a no.

29
00:02:34.469 --> 00:02:45.075
The trust curve then asks, for every threshold from 0.5 up to 0.99, what share of items clear it and how accurate the model is inside that share.

30
00:02:46.275 --> 00:02:47.116
What came back

31
00:02:47.716 --> 00:02:54.877
523 items got an answer and 21 came back as refusals, all of them hearsay fact patterns.

32
00:02:55.157 --> 00:02:58.195
Overall accuracy was 83.2%.

33
00:02:58.475 --> 00:03:06.518
The ranking was good: an AUROC of 0.894, and above 0.95 on two of the six tasks.

34
00:03:06.798 --> 00:03:16.853
The calibration was middling: an expected calibration error of 0.103 over ten bins and a Brier score of 0.140.

35
00:03:17.453 --> 00:03:20.699
Then the answer sheet, which is the point of the exercise.

36
00:03:21.299 --> 00:03:26.041
Each row is one question from the quiz, with the model's answer and how sure it was.

37
00:03:26.321 --> 00:03:34.603
The red crosses are the answers the lawyers marked wrong, and they are still there once only the answers it was 90% or more sure about are kept.

38
00:03:35.203 --> 00:03:36.671
The curve behind it is flat.

39
00:03:36.951 --> 00:03:48.564
Keep only the answers the model was at least 90% sure about and accuracy rises from 83% to 88%, in exchange for dropping 28% of the items.

40
00:03:48.844 --> 00:03:54.635
Push the bar to 99%, which still clears half the items, and it reaches 90%.

41
00:03:54.915 --> 00:04:05.414
There is no knee in the curve, no point where the confident set suddenly gets clean. 47 wrong answers sit inside the set the model was 90% sure about.

42
00:04:06.614 --> 00:04:07.692
The misses have a shape

43
00:04:08.292 --> 00:04:17.984
The reliability diagram says where they live. 242 of the 523 items got a probability close to zero, a confident no.

44
00:04:18.264 --> 00:04:21.029
Lawyers had marked 17% of those as yes.

45
00:04:21.629 --> 00:04:31.550
On the three contract tasks the model's average probability on clauses that really were absent was between 0.01 and 0.08.

46
00:04:31.830 --> 00:04:37.869
On clauses that really were present it was between 0.46 and 0.78.

47
00:04:38.149 --> 00:04:41.657
It is far more willing to say a clause is not there than to say it is.

48
00:04:41.937 --> 00:04:48.729
On change of control it missed about half the clauses that were there, most of them with a probability under 0.1.

49
00:04:49.329 --> 00:04:52.271
In a review workflow that is the expensive direction.

50
00:04:52.551 --> 00:04:54.630
A false yes costs a reviewer a minute.

51
00:04:54.910 --> 00:04:58.099
A confident false no never reaches the reviewer at all.

52
00:04:59.299 --> 00:05:00.905
The same number, three meanings

53
00:05:01.505 --> 00:05:04.629
Confidence meant different things depending on what I asked.

54
00:05:04.909 --> 00:05:11.933
On personal jurisdiction, the model cleared the 90% bar on only 8 of 50 items, and all 8 were right.

55
00:05:12.213 --> 00:05:20.147
On the contract tasks it cleared the bar on about four items in five, and between one in ten and one in four of those were wrong.

56
00:05:20.427 --> 00:05:27.068
On consumer contracts it cleared the bar on 87 of 100 and was right on 98% of them.

57
00:05:27.668 --> 00:05:30.475
So a single global threshold is the wrong tool.

58
00:05:30.755 --> 00:05:36.316
0.9 is a conservative reviewer on one task and a careless one on the next.

59
00:05:36.596 --> 00:05:41.975
The number only means something once you have measured it against your own labelled set, task by task.

60
00:05:43.175 --> 00:05:44.082
The refusals

61
00:05:44.682 --> 00:05:52.127
21 of the 94 hearsay fact patterns, 22%, came back as a refusal instead of a probability.

62
00:05:52.407 --> 00:06:04.300
Hearsay hypotheticals in a law exam describe assaults, thefts and threats, which is what trips a content filter. 18 of the 21 refused items were ones the lawyers had marked no.

63
00:06:04.900 --> 00:06:07.145
That matters for the pipeline more than the model.

64
00:06:07.425 --> 00:06:09.897
If a refusal is routed to a person, fine.

65
00:06:10.177 --> 00:06:15.559
If it is treated as a no, or silently dropped, the error rate on that task roughly doubles.

66
00:06:15.839 --> 00:06:23.942
It also means the hearsay accuracy above is measured on the 73 items the model was willing to read, not all 94.

67
00:06:25.142 --> 00:06:26.648
The prompt is part of the test

68
00:06:27.248 --> 00:06:28.608
One change from the plan.

69
00:06:28.888 --> 00:06:38.159
LegalBench's own instruction for hearsay is a single line: "Hearsay is an out-of-court statement introduced to prove the truth of the matter asserted.

70
00:06:38.439 --> 00:06:39.511
Is there hearsay?"

71
00:06:39.791 --> 00:06:45.979
In a ten-item pilot the model answered yes on all seven items marked no, and got one of nine right.

72
00:06:46.259 --> 00:06:51.778
It seemed to be answering "is there an out-of-court statement", which almost every fact pattern has.

73
00:06:52.378 --> 00:07:06.003
For the full run I gave it a four-sentence statement of the rule instead: in-court statements are not hearsay, conduct is only a statement if intended as one, a statement offered for something other than its truth is not hearsay.

74
00:07:06.283 --> 00:07:09.326
On the same ten items that went from one right to three.

75
00:07:09.606 --> 00:07:12.550
On the full 73 it reached 74%.

76
00:07:12.830 --> 00:07:18.647
The API answers the question you wrote, not the one you meant, and the difference showed up as 30 points.

77
00:07:19.847 --> 00:07:20.866
What I would do with it

78
00:07:21.466 --> 00:07:25.471
As a first pass that ranks a review queue, it is useful and nearly free.

79
00:07:25.751 --> 00:07:39.806
An AUROC of 0.89 across six legal tasks for two cents is a real result, and on the contract tasks the ordering is good enough that a reviewer starting from the top would find most of what matters early.

80
00:07:40.406 --> 00:07:55.070
As a thing that decides on its own above some threshold, it is not ready for questions like these. 12% wrong on the confident 72% is not a number a legal review lead would sign, and the errors skew toward missing things.

81
00:07:55.670 --> 00:07:57.778
The usable pattern is asymmetric.

82
00:07:58.058 --> 00:08:01.828
Trust a confident yes, since the model rarely invents a clause.

83
00:08:02.108 --> 00:08:05.298
Treat a confident no as unverified rather than clear.

84
00:08:05.578 --> 00:08:12.604
Set the threshold per task, from your own labelled sample, and count refusals as review items with a cost of their own.

85
00:08:13.204 --> 00:08:16.092
The question it leaves open is the one I started with.

86
00:08:16.372 --> 00:08:19.180
If the probability is not the trust signal, what is?

87
00:08:19.460 --> 00:08:21.143
A second model that disagrees?

88
00:08:21.423 --> 00:08:24.521
A per-task calibration table maintained by hand?

89
00:08:24.801 --> 00:08:28.167
Or just the old answer, a person, with a better-sorted queue.

90
00:08:29.367 --> 00:08:30.605
What this does not show

91
00:08:31.205 --> 00:08:37.503
One model, one call per item, zero-shot, with the benchmark's own question as the only instruction.

92
00:08:37.783 --> 00:08:42.171
A tuned prompt or a few examples would move these numbers, probably a lot.

93
00:08:42.451 --> 00:08:44.275
The hearsay rewrite shows how far.

94
00:08:44.875 --> 00:08:47.105
50 to 100 items per task.

95
00:08:47.385 --> 00:08:52.340
A per-task accuracy here carries a margin of roughly seven to ten points either way.

96
00:08:52.620 --> 00:08:55.353
The overall numbers are steadier than the task ones.

97
00:08:55.953 --> 00:08:58.806
LegalBench items are short and self-contained.

98
00:08:59.086 --> 00:09:03.934
Real contract review means long documents, defined terms and cross-references.

99
00:09:04.214 --> 00:09:05.670
This says nothing about that.

100
00:09:06.270 --> 00:09:07.898
The labels are the benchmark's.

101
00:09:08.178 --> 00:09:12.943
Lawyers disagree with each other too, and I did not audit the items the model got wrong.

102
00:09:13.543 --> 00:09:19.600
Nothing here compares the Decisions API to the Responses API or to any other model.

103
00:09:19.880 --> 00:09:24.987
It measures one thing: whether the probability this endpoint returns can be trusted on its own.

104
00:09:26.187 --> 00:09:27.206
Data and code

105
00:09:27.806 --> 00:09:34.766
The scripts, the prompts, the per-item scores and the curve as a table are in the repository on GitHub, linked here.

106
00:09:35.046 --> 00:09:38.048
Five commands reproduce the run for about two cents.

107
00:09:38.328 --> 00:09:45.144
Legal Bench is by Guha and others, 2023, released under a Creative Commons attribution licence.