Before you move into tactical conversations about evaluating legal AI, you’ve got to make sure all your stakeholders are speaking the same language.
This sounds like a given, right? But in practice, a surprising number of AI conversations jump right off the rails because of inconsistent lingo. A litigator asking whether “the AI” is “reliable” and a vendor explaining that “the model performs well” may both be speaking in good faith, but about entirely different things.
For example, here’s a common vocabulary problem watering down AI conversations all over the place: the words “capable,” “reliable,” and “accurate” are often used as though they mean roughly the same thing in the context of AI and its suitability for legal tasks.
The trouble, of course, is that they don’t all mean roughly the same thing. For legal teams making high-stakes decisions about which AI tools to trust, how to evaluate them, and how to defend their use when someone asks, the differences matter quite a bit.
A shared vocabulary can help you shape questions, understand and challenge their answers, and take ownership of the accountability that still falls on humans if something goes wrong. Let's sort out these terms to help arm you with the right lingo for your next AI conversation.
Accurate: AI’s Answer is Correct
Accuracy is the most straightforward of the three. It means correctness: freedom from mistakes or errors. An AI that gives you a factually correct, well-sourced answer to a specific question has demonstrated accuracy on that question.
Naturally, that’s pretty useful to know. But, upon closer examination, it’s also a relatively low bar because it only describes a single output in a single moment.
The question accuracy answers is: is this output right or wrong?
Capable: The AI Can Serve Your Use Case
Capability builds on accuracy. A tool that can produce accurate outputs – correct answers, sound analysis, properly cited references – for a particular type of work has demonstrated that it is capable of the task you’ve given it. In other words, it is capable of doing that thing accurately.
Aurora Bryant, senior manager of applied legal intelligence at Relativity, draws the line here cleanly: “Capability is the question of whether the tool can do the thing. A Q&A model that produces a correct, well-cited answer one time has demonstrated capability.”
Demonstrating capability once is meaningful. It tells you the system is worth paying attention to. However, it doesn’t necessarily tell you whether you can count on it to perform well under your circumstances over time.
Reliability: AI Can Be Trusted to Support Legal Practice at Scale
This is where reliability comes in.
Reliability is defined not by what a tool can do, but by whether it does it consistently, at the volume your work demands, within the margin for error your use case can tolerate, and under varying conditions.
Some further thoughts from Aurora: “If you ask that same question in 50 separate attempts and receive an equally confident but factually inaccurate answer 50 percent of the time, that tool has a serious reliability problem.”
That’s the tricky part, isn’t it? “Equally confident.” AI needs careful evaluation and validation because its answers appear credible, sound, and well-put-together even when they’re completely wrong. While more evident errors can and do happen, the greater risk for legal teams occurs when a system fails invisibly, behaving the same whether it’s factually correct or not.
So: if an AI can give you an accurate answer to a question, that shows capability. If it can do that consistently, at volume, within the standards your use case requires – that’s reliability. Reliability is what you build your practice on, and no one can evaluate an AI for reliability in your context quite like you can.
Why the Distinction Changes How You Evaluate
An intentional definition of terms – which should include, but isn’t limited to, capability, reliability, and accuracy – lends precision to the questions you ask when evaluating a particular AI system. You should be asking these questions of vendors, of tools you're already using, and of yourself when something produces a result that surprises you.
When a vendor shows you a demo or impressive benchmark data, they’re showing you capability by demonstrating what the system can do under good conditions, with well-chosen inputs. This is a great starting point, but remember: it isn’t automatically evidence of the tool’s reliability for your team, specifically.
To evaluate that, you’ll need to conduct some kind of validation exercise on your own data, for your own matter type, under your own conditions. Here, reliability becomes discernable and measurable.
What that exercise looks like isn’t set in stone; there are so many evaluation methodologies out there, each requiring different methods, rigor, and risk tolerances. A lot depends on where the particular use case you’re evaluating – because you can really only measure one at a time – falls on the spectrum from low-stakes drafting assistance to court-facing production.
Evaluation workflows take many forms. Get practical guidance on some of your options in this e-book.
What "Reliable AI" Means at Relativity
For us, reliability isn't just a talking point — it's a design commitment and an evaluation standard that shapes how we build and ship.
Every AI model we develop goes through extensive peer review and validation against representative customer use cases before it reaches your hands. We don't ask whether a model can perform well on a benchmark. We ask whether it performs consistently, defensibly, and at the level your work actually requires — across the types of data you encounter, the matter types you handle, and the standards you'll be held to if your process is ever scrutinized.
That's what "reliable" means to us in practice. Capable of the task, yes — but consistently capable, at scale, in your context, with documentation to back it up.
It's a standard we hold ourselves to, and increasingly, it's the standard we'd encourage you to hold every AI tool to — including ours.
The Question Worth Asking
The next time you're evaluating an AI tool — or fielding a question from a client or colleague about one you're already using — it's worth pausing on which of these three things you're actually talking about.
Is this a question about accuracy? One output, one moment, one check.
Is it about capability? What the system can do when conditions are right.
Or is it about reliability — what the system does consistently, at scale, under the conditions your practice actually operates in?
That last question is harder to answer. It requires more rigorous evaluation, more specificity about your use case, and more honesty about the gap between a vendor's demonstrated capability and your team's real-world needs. It also happens to be the most important question legal teams can ask right now.
If you want a practical framework for working through it, AI Trust in Practice: How Legal Teams Evaluate What Works is a good place to start. We built it with just this conversation in mind.
Graphics for this article were created by Sarah Vachlon.
