There's something genuinely strange about the way a chatbot gets things wrong. Ask a person a question they don't know the answer to, and usually you can tell – they hesitate, qualify, admit the gap. Ask a large language model the same question, and it will likely answer fluently and completely. In the exact tone and register you'd expect from someone who actually knows.
The problem isn't that AI systems are confidently wrong some of the time. It's that there's often no signal in the response itself to tell you when that's happening.
That's a core challenge we think about at Relativity when it comes to aiR Assist – and it's one of the reasons evaluation is as much a part of what we build as the product itself.
Why AI answers look right even when they're not
Most people have a rough mental model of how AI assistants work: you ask a question, the model searches the data on which it was trained, and gives you an answer. But that's not quite right.
Large language models don't retrieve data the way a database does. Instead, they generate text one token at a time, predicting what’s most likely to come next based on patterns learned from vast amounts of human-written text. That produces fluent, coherent output, but it also means the model has no built-in way to distinguish between something it actually "knows" and something it has plausibly constructed.
When the model's footing is uncertain, the output doesn't necessarily reflect that uncertainty. It continues in the right vocabulary, the right structure, the right tone. A model trained on legal documents learns to write like a legal document. That means when it fills a gap, it does so convincingly.
These confident gap-fills are hallucinations, and they’re a well-understood, predictable feature of how generation works. Predictable, in this case, means testable: we know what to look for. But that's a separate question from what the model itself will tell you. The surface of the response – fluent, well-formatted, assured – will never flag the problem on its own.
The standard mitigation is Retrieval-Augmented Generation, or RAG. Rather than relying entirely on training, a RAG system first pulls relevant documents from a defined corpus (in aiR Assist's case, the actual case record) and then uses the model to synthesize an answer grounded in that retrieved material. The citations throughout aiR Assist’s responses tether the answers to real evidence.
Because we know exactly where the residual risk sits, our evaluation suite is built specifically around it: the places where a model working with real documents might still misrepresent what those documents say, assert something the record doesn't cover, or attach a citation to a claim it doesn't actually support. Those are manageable, well-defined failure modes, and with aiR Assist, we've built specific checks for each one.
Why standard validation metrics don’t apply to AI assistants
When we first start thinking about how to evaluate an AI tool in a legal context, the instinct is usually to reach for the metrics we already know: precision and recall. Those are the workhorses of document review assessment – they tell you how many relevant documents the system found, how many it missed, and how much extra the system added. They're well understood, defensible, and in the context of review, the right thing to measure.
Relativity aiR for Review is built around those metrics, and appropriately so. When you're evaluating a system that's classifying documents as responsive or non-responsive, precision and recall are exactly what you care about. There's a defined corpus, a ground truth, a set of decisions with right answers. You can measure how well the system performs against that standard.
aiR Assist, however, does something fundamentally different. It produces written answers to natural-language questions – answers with citations, analysis, and synthesis across multiple documents. There's no single ground truth to score against, no fixed set of retrieved documents to evaluate coverage over, and at production scale, there's no exhaustively labeled corpus to compare against in the first place.
But more importantly, even if that data existed, no precision/recall score could tell you whether an answer inverted an attribution, stripped the governing exception from a qualified finding, or sounded authoritative about something the record simply doesn't establish.
The failure modes that matter for aiR Assist aren't retrieval failures. They're interpretation failures, faithfulness failures, and reasoning failures. They’re more nuanced than hit-or-miss retrieval errors, but once you know what to look for, they’re just as traceable.
Building the right evaluation framework for aiR Assist meant taking the same commitment to rigor that underpins aiR for Review and applying it to a harder problem. And that’s just what we did.
What we check in every aiR Assist answer
The evaluation suite we built runs every aiR Assist answer through more than a dozen independent checks. The team behind it includes former practicing lawyers with decades of federal litigation experience working alongside applied scientists. That combination is crucial, because the failure modes we’re testing for are legal failure modes, not just statistical ones, such as:
Faithfulness to the record
Does the answer preserve what the documents actually say – not just roughly, but in terms of strength, scope, and qualification? Separate checks catch things like paraphrases that shift a denial into something more vague, excerpts that drop the conditions governing a finding, and hedged conclusions presented as certainties. Legal meaning often lives in precisely those distinctions, and aiR Assist is built to preserve them.
Groundedness
Every claim needs to be substantiated by the retrieved record – either stated outright or genuinely entailed by it. A claim that sounds plausible given the general context doesn't qualify. Neither does a confident statement about something the record is simply silent on. Where the evidence doesn't speak, the answer has to say so rather than fill in the gap. This is the check that most directly catches the hallucination behavior described above, and it runs on every response.
People, authorship, and time
Real litigation data is messy in predictable ways. Long email chains, similarly named individuals, documents that span years of evolving relationships and corporate structures. We check that the system isn't conflating entities or inverting relationships (who acquired whom, which lawyer represents which party); that attribution tracks to who actually wrote something rather than who forwarded it; and that timestamps aren’t confused. Sent, received, and forwarded dates can tell very different stories.
Citation quality
This breaks into four distinct checks: whether every factual claim carries a citation, whether citations follow a consistent and parsable format, whether every source that contributed is cited without phantom references to nonexistent documents, and – critically – whether each citation actually supports the specific claim it accompanies. That last one is the hardest to catch and the most consequential to miss, which is why it has its own dedicated check in the suite.
Completeness
Multi-part questions need every part answered. Material retrieved content shouldn't be silently dropped. Names, dates, and figures should appear in their most complete and precise form. In legal work, what's left out can matter as much as what's included, and completeness checks are built around that reality.
How the testing in aiR Assist actually works
Evaluation runs against both synthetic, expert-annotated data sets and real production-scale litigation corpora – collections that reach into the tens of thousands of documents. The evaluation suite covers dozens of distinct test scenarios, each expanded into hundreds of question variations, each scored against multiple pass/fail criteria. Total evaluation points run well over a thousand.
The methodology is deliberate about strictness. Evaluator judgments have to be backed by verbatim excerpts from the record – not paraphrases, not general impressions. We hold answers to the same standard a litigator would: one wrong attribution results in a critical error flag, regardless of what else the answer got right.
We also separately measure consistency across repeated runs of the same question. A system that gives substantively different answers to the same question across different queries is unreliable even if each individual answer looks reasonable. That kind of variance is easy to miss if you're only spot-checking outputs.
Moreover, our evaluation started well before we released the product. But it didn’t stop there. The suite runs continuously as the underlying models, prompts, and retrieval components evolve. Any change to any part of the system gets evaluated against the full battery before it ships. This is a continuous standard, and it’s what gives us the confidence to say that aiR Assist’s reliability isn’t just a snapshot from launch day.
High stakes, higher bar
The way we think about aiR Assist evaluation is ultimately a reflection of how we think about deploying generative AI in a context where mistakes have real consequences. Consumer AI tools and professional legal AI tools share underlying technology but operate in categorically different risk environments. The stakes of a wrong answer, the domain knowledge required to catch one, and the downstream effects of missing it are all different in kind – and the evaluation infrastructure has to reflect that.
Designing evaluation that's adequate for that environment required embedding legal expertise in the reliability engineering process – not just as a reviewer at the end, but also as a core part of how the checks are defined and refined. It required resisting the pull toward metrics that are easy to compute but don't capture what actually matters. And it required treating evaluation as ongoing infrastructure rather than a one-time assessment.
The underlying principle is that in a high-stakes domain, reliability has to be engineered – not assumed and not retrofitted after the fact. That's what the aiR Assist evaluation suite represents: the ongoing infrastructure behind a product built, from the ground up, to be trusted.
Graphics for this article were created by Kael Rose.
