Retrieval at scale · regulated profession
Making 20 million judgments answer a lawyer's question
The case that wins the argument exists — buried under twenty million others, half of them no longer good law.
§01The problem
A lawyer's question is never 'find documents about X'. It's 'find the authority that survives scrutiny': the judgment that is on point, from a court that binds this one, that hasn't been overruled since. Twenty million judgments deep, that's a needle in a stack of needles — and citing a dead case in court isn't an inconvenience, it's professional damage.
Generic RAG fails here in ways that matter: it retrieves the eloquent paragraph over the authoritative one, misses the exact case the lawyer named because embeddings blur specifics, and cheerfully cites decisions that were overruled a decade ago.
§02The thinking
The breakthrough framing was that legal search is three different products pretending to be one: known-item lookup ('give me exactly this case'), open research ('what's the law on X'), and validation ('is this still good law'). Each has different failure costs, so each earned a different retrieval path.
Known-item lookup needed deterministic behaviour — a lawyer typing a famous case name must get that case, rank one, every time; embeddings alone can't promise that. Open research needed authority-aware ranking, where the citation graph — who cites whom, and how courts treated the citation — outranks textual similarity. Validation needed the treatment history made explicit: followed, distinguished, overruled.
Hallucination policy was absolute: every proposition pinned to a real, checkable citation, and 'no strong authority found' is a legitimate answer — in law, a confident wrong answer is the worst product outcome available.
+ Three retrieval paths under one search box
− one embedding pipeline for everything
# a system that's 90% right on all three intents is worse than one that's near-perfect on lookup and honest about the rest.
+ Citation-graph authority as a ranking signal
− pure semantic similarity
# in law, what other courts did with a judgment matters more than how well its prose matches the query.
+ Pin-cited, verifiable answers
− fluent synthesis
# the product's job is to be checkable in the format lawyers already trust — paragraph and citation, not paraphrase.
§03What shipped
The shipped system routes each query by intent: exact case names hit a deterministic lookup lane; research questions flow through retrieval ranked by the citation graph and court hierarchy; every result carries its treatment history, so 'still good law?' is answered before it's asked. Answers cite pinpoint paragraphs a lawyer can verify in seconds.
§04Outcomes
Named case, rank one
known-item lookups behave deterministically — the trust floor
Authority over eloquence
results ranked by how courts treated a judgment, not how it reads
Overruled means flagged
treatment history travels with every citation
§05Reflection
This domain taught me that retrieval quality is a product decision wearing an engineering costume. Choosing what must never fail (the named case), what may gracefully hedge (open research), and what must be brutally explicit (validity) shaped the architecture more than any model choice.
$ cat takeaways.txt
- ✓ Split retrieval by intent and by cost-of-failure, not by technique.
- ✓ Domain authority signals beat general similarity in specialised corpora.
- ✓ In regulated domains, 'not found' is a feature. Confident fabrication is the only true failure.