Six of 25 answers never came back from search.
6 of 25 never retrieved. Recall 0.760.
Notes
Notes
One counted finding per note, with its denominator, its figure and the file the numbers come from. The short version of what I post on LinkedIn and X.
-
crosssource · 25 questions · retrieval
-
cheap-tier-legal-tax-hr · 1,414 cases
Where a 10-cent model goes in tax, legal, HR
One of four can say when it is unsure.
-
screener-eval · one resume · two models
One resume, one posting: 28 and reject, 79 and advance.
Mean gap 22.3 points over 21 postings.
-
decisions api · 544 questions · one run
At 0.9 confidence, 47 of 377 were wrong.
47 of 377 wrong. No threshold hit 95%.
-
mirror-eval · 63 probes · two waves
Claude cited my site on 0 of 63 probes. Twice.
0 of 63 in August. 0 of 63 in September.
-
decisions api · hearsay · 94 items
21 of 94 hearsay items came back refused.
21 refused. 18 of them were a no.
-
mirror-eval · judge check · 40 labels
Two judges, 40 blind labels. Matches: 3 and 12.
3 of 40 and 12 of 40 matched.
-
recruiter-prompt-bank · 28 sep 2026
444 collected, 239 kept. The 205 rejects are public.
20 of 444: the page did not hold the text.
-
screener-eval · four edits · n = 21
Two levers that did not move. One that did.
Largest edit effect 1.14. Screener swap: 22.3.
-
mirror-eval · 11 predictions
I wrote 11 predictions before the run. Three missed.
3 of 11 missed. 4 held.
-
decisions api · the near-certain noes
242 answers at 0.1 or less. The lawyers said yes to 41.
41 of 242 near-certain noes were a yes.
-
screener-eval · one pdf · five parsers
Three parsers lost the same three fields. The links.
26 of 29, three times. Then 29 of 29.
-
crosssource · judge check · 15 labels
Same default model writes and judges. 15 labels.
15 of 15 agreed. Interval floor 0.78.
-
mirror-eval · claude · two waves
Poisoned sources: 79 to 11. A data broker: 0 to 74.
ZoomInfo: 0 to 74 citations.
-
nobody reports the misses · six cases
Wrong answers get caught. Missed items do not.
Six kinds of work. Six ways to count a miss.
-
mirror-eval · perplexity · 29 days
13 of 63, then 63 of 63. A cluster of changes.
13 to 63 of 63 in 29 days.
-
screener-eval · verdicts · two models
One screener's default was reject. The other's was hold.
Reject 136 of 233. Hold 157 of 232.
-
crosssource · two prompts · n = 25
Two prompts, 25 questions: two fewer wrong passages.
Wrong passage: 3 of 143, then 1 of 142.
-
recruiter-prompt-bank · 20 vendors
What AI screeners say they check: 14, 9, 7, 4, nothing.
4 of 20: no criteria, or not disclosed.
-
tradeframe · 438 runs · all green
One run took 59 ms. The log said success.
9 silent failures behind 438 green runs.
-
mirror-eval · prediction 9 · agreement
I set the floor at 60%. Agreement came in at 35.
117 of 332 answers agreed. 35%.
-
screener-eval · five reps · the noise
Same cell, two runs: 62.0 and 47.2. Five calls each.
Noise 3.0 and 4.8 points. Five reps run.
-
decisions api · 224 answers · certain
224 answers at exactly 0 or 1. Eighteen were wrong.
18 of 224 wrong at a probability of 0 or 1.
-
recruiter-prompt-bank · 239 records
Labels: hiring manager 3 of 239, vendor 173.
173 of 239 in the vendor's voice.
-
decisions api · 544 calls · cost
The eval cost two cents. The day was the budget.
190,399 input tokens. $0.019.
-
portfolio · the build gate
My site fails CI unless it still says 74 things.
74 required. 11 banned. Exit 1 otherwise.
-
decisions api · hearsay · two prompts
The benchmark's one-liner: 1 of 9. Four sentences: 3 of 8.
Same ten items. 1 of 9, then 3 of 8.
-
recruiter-prompt-bank · one counter
My own counter was wrong. Link is inside LinkedIn.
34 of 239 by substring. 17 by word.