crosssource · 25 questions · retrieval
Six of 25 answers never came back from search.
The note
Twenty-five question cards fly, one at a time, to a shelf of 22 court opinions. Six come back empty.
On 6 of the 25 questions in my citation eval, the one passage that holds the answer was not in the top five the retriever handed to the model.
I ran two prompt configurations and they agreed: citation recall 19 of 25, 0.760, in both runs. Retrieval is upstream of the prompt and returned the same lists both times.
Underneath, the generator could not cite what it never saw. Question g05 asks what rule governs the law a federal court applies in diversity cases. Retrieval returned chunks 0, 1, 3 and 26 of Erie Railroad v. Tompkins and chunk 28 of Marbury v. Madison, but not chunk 6 of Erie, the one that holds the answer. The model stated the rule anyway, from chunk 3, which paraphrases it. Recall for that item still scores zero, because the key names one chunk and the answer cited another.
The fix was never going to be in the prompt.
Limits. 22 public-domain opinions cut into 463 chunks at a target of about 350 words, median 292. BM25, top five, no reranking. Twenty-five questions written by a non-lawyer, me. All of it directional, none of it significant.
When the answer states the right rule from the wrong chunk and the key scores it zero, which do you fix: the key, the retriever, or neither?
Where the numbers live
The harness, the 25 questions and the six that never came back
22 opinions, 463 chunks, one non-lawyer's questions. Directional, not significant.