CrossSource: measuring whether a RAG system can be trusted to cite the law
Strict citation prompting lifted citation precision from 0.981 to 0.994 on 25 questions; recall stayed at 0.760 because retrieval, not the model, was the limit. CrossSource is the open harness behind that number: when a legal RAG system cites a court opinion, does the opinion support the claim?
CrossSource v0.1, Table 2 below. Axis starts at 0.70.
Listen to this piece · 6 min
0:00 / 6:22 · resumes at 0:00
Read by a synthetic voice (Kokoro).
Why this exists
In Mata v. Avianca, lawyers were sanctioned for filing a brief full of citations an AI invented. Citation hallucination is the failure legal and tax AI most has to stop before a practitioner relies on an answer. It is the same failure I work on in production. CrossSource is the public, reproducible version of that work: a dataset where every item traces to an exact passage in a public opinion, a written labelling protocol, and an evaluation pipeline anyone can run.
Method
- v0.3 dataset
- 150 items: 75
qa_citation, 75 citation_check: over 45 public US opinions from CourtListener (27 federal circuit, 18 state supreme court, 2018 or later), across 15 legal topics. - Provenance
- Every item carries its opinion ID, URL, the exact passage and the cited authority.
- Labelling
- A written protocol,
LABELING_PROTOCOL.mdv1.1. Items are model-drafted and model-checked; no human labelled every item. - Agreement
- Two blind, independent model raters: Cohen's kappa 0.98 on citation_check, 0.84 on qa_citation.
- Adjudication
- Every disagreement plus a 10% audit sample, 30 items in all, adjudicated by me with a third independent model rater. 27 of 30 matched the recommended label.
- v0.1 corpus
- 22 public-domain US court opinions (16 Supreme Court, 2 federal circuit, 4 state), sourced from Harvard Law School's Caselaw Access Project, chunked to ~350 words.
- v0.1 retrieval
- BM25, top-5.
- v0.1 generation
- Claude with mandatory inline citations in
doc_id:chunk_idformat. - v0.1 golden set
- 25 human-authored questions with verified ground-truth supporting chunks.
- v0.1 configurations
- baseline (citation formatting only) vs strict (explicit citation-discipline rules in the prompt).
- v0.1 scoring
- claim-level citation precision and recall, faithfulness, and answer relevance: scored by an LLM judge, validated by hand (below).
Findings
| Model | Citation precision | Citation recall | citation_check accuracy |
|---|---|---|---|
| claude-sonnet-5-5 | 0.919 | 0.718 | 1.00 |
| gpt-5-mini | 0.763 | 0.718 | 0.93 |
Baseline prompt9 failures
Strict prompt7 failures
- Citation precision
- 0.981 → 0.994
- Citation recall
- 0.760 → 0.760
The strict prompt had 2 fewer wrong-passage failures than baseline (3 → 1). The 6 missing-authority failures were the same in both configurations and trace to retrieval.
Source: Tables 2 and 3 below (v0.1, n = 25). Wrong-passage failures fell under the strict prompt; missing-authority failures trace to retrieval.
| Dimension | Baseline | Strict |
|---|---|---|
| Citation precision | 0.981 | 0.994 |
| Citation recall | 0.760 | 0.760 |
| Faithfulness | 1.000 | 1.000 |
| Answer relevance | 0.980 | 0.960 |
| Error type | Baseline | Strict |
|---|---|---|
| Right document, wrong passage | 3 | 1 |
| Missing authority | 6 | 6 |
| Wrong document | 0 | 0 |
| Unsupported claim | 0 | 0 |
The v0.1 finding that still stands (n = 25): citation-discipline prompting buys precision: wrong-passage citations drop 3 → 1, precision rises 0.981 → 0.994. It cannot buy recall: 0.760 in both configurations, capped by retrieval, because every missing-authority failure traces to retrieval, not generation. Knowing which layer owns the failure is the entire point of separating the metrics.
The judge caught a bug, because I checked the judge
In v0.1, I hand-graded a blind, stratified sample of the judge's citation-precision verdicts:
100% agreement (15/15).
That validation pass also surfaced a genuine harness defect: consecutive citations produced punctuation-only claim spans that the judge initially mislabeled as failures. Fixing it changed the metrics.
Transcript
- The answer. The model answers a legal question and cites its sources inline.
- Claim spans. The harness cuts the answer into claim spans. The judge will check each one against the passage it cites.
- The bug. Where two citations sit side by side, a span is left holding nothing but punctuation, a fragment like "." that is not a claim.
- The judge scores it. The judge dutifully scores that span as a citation failure. A harness bug now looks like a model failure.
- The blind check. I hand-graded a blind, stratified sample of the judge's verdicts: 15 of 15 agreed. Checking the judge is what exposed the bug.
Source: this section and the judge essay. The spans drawn are placeholders, not real answer text. Read the essay →
That is the lesson worth paying for: an unvalidated eval reports wrong numbers with full confidence, precisely the failure mode the eval exists to catch.
What this demonstrates
The same discipline I use on production systems and on anything I build:
- Instrument the pipeline
- Keep provenance for every item, so any score traces back to a passage
- Build a failure taxonomy instead of a single score
- Check the judge against people before trusting it
- Separate precision from recall, so you know whether to fix the prompt or the retrieval
Release (v0.3)
Related writing
AI PRODUCT MANAGER · AGENTS, WORKFLOWS AND EVALS
Read the code, run the harness