CrossSource: measuring whether a RAG system can be trusted to cite the law
An open evaluation harness for citation accuracy in legal RAG — built to answer one question with rigor: when the system cites a court opinion, is the claim actually supported by that source?
In Mata v. Avianca, lawyers were sanctioned for filing a brief full of citations an AI invented. Citation hallucination is the canonical trust failure of legal AI — and it is exactly the class of failure my day job revolves around. CrossSource is the public, reproducible version of that work: a fixed corpus, a golden question set, and an evaluation pipeline anyone can run.
- Corpus
- 22 public-domain US court opinions (16 Supreme Court, 2 federal circuit, 4 state), sourced from Harvard Law School’s Caselaw Access Project, chunked to ~350 words.
- Retrieval
- BM25, top-5.
- Generation
- Claude with mandatory inline citations in
doc_id:chunk_idformat. - Golden set
- 25 human-authored questions with verified ground-truth supporting chunks.
- Two configurations
- baseline (citation formatting only) vs strict (explicit citation-discipline rules in the prompt).
- Scoring
- claim-level citation precision and recall, faithfulness, and answer relevance — scored by an LLM judge, validated by hand (below).
| Dimension | Baseline | Strict |
|---|---|---|
| Citation precision | 0.981 | 0.994 |
| Citation recall | 0.760 | 0.760 |
| Faithfulness | 1.000 | 1.000 |
| Answer relevance | 0.980 | 0.960 |
| Error type | Baseline | Strict |
|---|---|---|
| Right document, wrong passage | 3 | 1 |
| Missing authority | 6 | 6 |
| Wrong document | 0 | 0 |
| Unsupported claim | 0 | 0 |
The finding that matters: prompting discipline buys precision — wrong-passage citations drop 3 → 1, precision rises 0.981 → 0.994. It cannot buy recall: 0.760 in both configurations, because every missing-authority failure traces to retrieval, not generation. Knowing which layer owns the failure is the entire point of separating the metrics.
I hand-graded a blind, stratified sample of the judge's citation-precision verdicts: 100% agreement (15/15).
That validation pass also surfaced a genuine harness defect — consecutive citations produced punctuation-only claim spans that the judge initially mislabeled as failures. Fixing it changed the metrics.
That is the lesson worth paying for: an unvalidated eval reports wrong numbers with full confidence — precisely the failure mode the eval exists to catch.
The same discipline I run in production:
- Instrument the pipeline
- Build a failure taxonomy instead of a single score
- Validate the judge externally
- Separate precision from recall — so you know whether to fix prompting or retrieval
In build
- Retrieval ablations
- Cross-model judge comparison
- Law-trained annotation review
- Per-question difficulty stratification
- An agent-eval module — in build
Read the code, run the harness →