FirmMemory
Governed retrieval over a law firm's matter archive. Built March to May 2026, self funded.
A private answer layer over a law firm's own work. FirmMemory finds relevant matters, connects evidence across the archive and builds sourced answers from what the firm has actually done. When the precedent is thin or absent, it says so.
Context
A firm's most valuable asset is what it has already done, and it cannot reach it. The question a partner actually asks, "have we advised on something like this, and what came up", is not a query keyword search can answer.
The thinking
Put a language model over that archive and it will answer. It will answer when the firm has done the work, and when it has not. In legal advice the second failure is far worse than returning nothing. So the design problem was never retrieval quality. It was behaviour at the edges of what the firm knows.
What I built
Provenance captured at ingestion and carried through synthesis, so every claim traces to its source paragraph. Gap detection is deterministic, not inferred: a full scan over each shortlisted matter, no similarity gate. I tried the similarity version first and it missed high signal paragraphs that scored badly against the query wording. A gap check a language model can talk its way past is not a gap check.
Refusal is designed behaviour, not a fallback. When evidence is thin the system says so, and says what it looked at. The corpus is 114 matters and 15,084 paragraphs, reconciled three ways so the index, the store and the source agree.
What I measured
A 30 query adversarial evaluation kit, six scoring axes, reproducible regression scripts, an invariant failure budget. The number that matters more than the score: identical runs varied by plus or minus one query. That is the noise floor. Any difference smaller than it is not a finding, and most published evaluations never establish one.
Why I removed the thinness gate
I designed a retrieval thinness gate to flag when evidence was too sparse to answer on, then calibrated it across 16 queries and four intent categories. Separation was near monotonic, so the mechanism was real. But no defensible threshold existed. Every cut point I could justify for one intent category was wrong for another.
So I removed it. The honesty requirement moved onto transparency fields the user reads and judges directly, rather than a number the system decides on their behalf. The debt was logged and frozen. Without that calibration the gate ships: it has a plausible mechanism, an intuitive story and a number attached, and it fails quietly in a product whose whole value is telling you when it does not know. The harness did not improve this system by raising a score. It told me which of my own ideas the evidence did not support.
The limit
The corpus is synthetic. It is realistic in structure, length and messiness, and the architecture and evaluation harness run against it exactly as they would against a real archive, but no firm has put its own matters through this. The results describe behaviour on a corpus I built, and that is the only claim I make for them.
