CrossSource: measuring whether a RAG system can be trusted to cite the law

Strict citation prompting lifted citation precision from 0.981 to 0.994 on 25 questions; recall stayed at 0.760 because retrieval, not the model, was the limit. CrossSource is the open harness behind that number: when a legal RAG system cites a court opinion, does the opinion support the claim?

Fig. 1 Which number moved. v0.1, n = 25. Hollow: baseline prompt. Filled: strict prompt.

CrossSource v0.1, Table 2 below. Axis starts at 0.70.

Why this exists

In Mata v. Avianca, lawyers were sanctioned for filing a brief full of citations an AI invented. Citation hallucination is the failure legal and tax AI most has to stop before a practitioner relies on an answer. It is the same failure I work on in production. CrossSource is the public, reproducible version of that work: a dataset where every item traces to an exact passage in a public opinion, a written labelling protocol, and an evaluation pipeline anyone can run.

Method

v0.3 dataset
150 items: 75 qa_citation , 75 citation_check: over 45 public US opinions from CourtListener (27 federal circuit, 18 state supreme court, 2018 or later), across 15 legal topics.
Provenance
Every item carries its opinion ID, URL, the exact passage and the cited authority.
Labelling
A written protocol, LABELING_PROTOCOL.md v1.1. Items are model-drafted and model-checked; no human labelled every item.
Agreement
Two blind, independent model raters: Cohen's kappa 0.98 on citation_check, 0.84 on qa_citation.
Adjudication
Every disagreement plus a 10% audit sample, 30 items in all, adjudicated by me with a third independent model rater. 27 of 30 matched the recommended label.
v0.1 corpus
22 public-domain US court opinions (16 Supreme Court, 2 federal circuit, 4 state), sourced from Harvard Law School's Caselaw Access Project, chunked to ~350 words.
v0.1 retrieval
BM25, top-5.
v0.1 generation
Claude with mandatory inline citations in doc_id:chunk_id format.
v0.1 golden set
25 human-authored questions with verified ground-truth supporting chunks.
v0.1 configurations
baseline (citation formatting only) vs strict (explicit citation-discipline rules in the prompt).
v0.1 scoring
claim-level citation precision and recall, faithfulness, and answer relevance: scored by an LLM judge, validated by hand (below).

Findings

Table 1. v0.3 sanity run on a 30-item sample (not a full run)
Model Citation precisionCitation recallcitation_check accuracy
claude-sonnet-5-5 0.9190.7181.00
gpt-5-mini 0.7630.7180.93
Fig. 1 Which layer owns the failure

Baseline prompt9 failures

Strict prompt7 failures

Citation precision
0.981 → 0.994
Citation recall
0.760 → 0.760

The strict prompt had 2 fewer wrong-passage failures than baseline (3 → 1). The 6 missing-authority failures were the same in both configurations and trace to retrieval.

Source: Tables 2 and 3 below (v0.1, n = 25). Wrong-passage failures fell under the strict prompt; missing-authority failures trace to retrieval.

Table 2. v0.1 (n = 25): baseline vs. strict
Dimension BaselineStrict
Citation precision 0.9810.994
Citation recall 0.7600.760
Faithfulness 1.0001.000
Answer relevance 0.9800.960
Table 3. v0.1 (n = 25): failure taxonomy by configuration
Error type BaselineStrict
Right document, wrong passage 31
Missing authority 66
Wrong document 00
Unsupported claim 00

The v0.1 finding that still stands (n = 25): citation-discipline prompting buys precision: wrong-passage citations drop 3 → 1, precision rises 0.981 → 0.994. It cannot buy recall: 0.760 in both configurations, capped by retrieval, because every missing-authority failure traces to retrieval, not generation. Knowing which layer owns the failure is the entire point of separating the metrics.

The judge caught a bug, because I checked the judge

In v0.1, I hand-graded a blind, stratified sample of the judge's citation-precision verdicts:

100% agreement (15/15).

That validation pass also surfaced a genuine harness defect: consecutive citations produced punctuation-only claim spans that the judge initially mislabeled as failures. Fixing it changed the metrics.

Fig. 2 One answer through the harness

Transcript

  1. The answer. The model answers a legal question and cites its sources inline.
  2. Claim spans. The harness cuts the answer into claim spans. The judge will check each one against the passage it cites.
  3. The bug. Where two citations sit side by side, a span is left holding nothing but punctuation, a fragment like "." that is not a claim.
  4. The judge scores it. The judge dutifully scores that span as a citation failure. A harness bug now looks like a model failure.
  5. The blind check. I hand-graded a blind, stratified sample of the judge's verdicts: 15 of 15 agreed. Checking the judge is what exposed the bug.

Source: this section and the judge essay. The spans drawn are placeholders, not real answer text. Read the essay →

That is the lesson worth paying for: an unvalidated eval reports wrong numbers with full confidence, precisely the failure mode the eval exists to catch.

What this demonstrates

The same discipline I use on production systems and on anything I build:

  • Instrument the pipeline
  • Keep provenance for every item, so any score traces back to a passage
  • Build a failure taxonomy instead of a single score
  • Check the judge against people before trusting it
  • Separate precision from recall, so you know whether to fix the prompt or the retrieval

AI PRODUCT MANAGER · AGENTS, WORKFLOWS AND EVALS

Read the code, run the harness