mirror-eval · 11 predictions
I wrote 11 predictions before the run. Three missed.
The note
3 of 11 predictions missed. They sit in a file dated 3 August, before my AI search audit ran, each with its threshold written down.
The scorecard, as the file records it: 4 held (one of them tied), 1 partly held, 1 inconclusive, 3 missed, 2 could not be tested with the data the run produced.
The first prediction in the file was the worst: my site appears in fewer than 25% of search probes' citations in the baseline wave. The baseline came back at 136 of 252 search probes citing it. Fifty-four percent. Missed by a factor of two.
P5 said a specific failure would show up in a quarter of the 40 verification answers; across all 332 answers the band was 0 to 10 percent, and only one of the two judges ever tagged it. P9 said the two judges would agree with each other 60 to 75 percent of the time; they agreed 35 percent of the time, and the file's own rule says that number means the taxonomy is underspecified, not that the engines are erratic.
Writing the thresholds first did one thing: it stopped me from calling 54 percent a surprise in either direction after the fact.
The limit is underneath the scorecard. Only P1 and P11 rest on plain counts. The others rest on judge tags, and the judges matched my blind labels on 3 of 40 and 12 of 40, so those scores carry the judges' error with them.
Should a scorecard count a prediction as held when the judge that scored it failed its own blind check?
Where the numbers live
Two predictions rest on plain counts. Nine rest on judge tags, and the judges failed their blind check.