crosssource · two prompts · n = 25
Two prompts, 25 questions: two fewer wrong passages.
The note
"The cited passage [erie_v_tompkins:0] only lays out the factual background and parties' contentions, not the holding about the federal court's obligation." That is my eval's judge failing one citation in an Erie Railroad v. Tompkins answer.
Under my baseline prompt there were 3 such right-case, wrong-passage citations across 143 judged. Under a strict prompt that tells the model to cite the passage that carries the claim, 1 across 142.
A check on the case name would have passed all three. The next citation in that same sentence pointed at chunk 1 and was judged supported, so the reader had the right passage one bracket later. The eval counts each citation on its own, and I think it should.
The headline numbers barely noticed. Citation precision, averaged per question, went 0.981 to 0.994. Recall did not move: 0.760 both times, because the misses there were upstream in retrieval. Nothing in either run cited the wrong case or made a claim with no support.
Net, the prompt change cut wrong-passage citations by two, out of about 140. It is not significant at 25 questions.
Limits. 25 questions I wrote myself. No temperature set. Generator and judge from one model family.
Is two fewer wrong passages on 25 questions enough to change a production prompt, or only enough to keep testing?
Where the numbers live
The harness and the two prompt configurations, with the per-question rows
25 questions, a non-lawyer's golden set, no temperature set. Directional. One model family writes and judges.