Making the Case for Active Learning
Nick Cole, director of litigation support at Foley & Lardner, was faced with a huge task. For one, a data set of this size was incredibly challenging to review. Plus, Foley & Lardner could not afford to miss any relevant documents, or else their client faced serious consequences with the government agency.
The firm’s white-collar team didn’t immediately see the benefits of active learning—linear review and search terms were successful for them in the past. However, Nick believed that the search terms reports they were using were not inclusive enough to find all the responsive documents.
So, the litigation support team walked them through the active learning workflow to illustrate the time they would save. They explained how the technology continuously learns from coding decisions and predicts which documents are most likely to be responsive—ultimately yielding a smaller document set to review, as well as more accurate results.
With those convincing arguments, the white-collar team decided to let Nick show them the benefits of active learning directly on this case’s documents.
Quickly and Accurately Reviewing 110M+ Documents
Nick’s team needed to complete this review as accurately as possible and within the government agency’s requirements. To accomplish this, Foley & Lardner created a novel workflow with active learning to get through the data set.
Nick implemented a workflow that split the documents into sets of 10 million documents with 38 analytics indexes per set. Some of the analytics sets were random samples of the whole universe, while other sets focused on particular subject matters. Different sets of exemplars were used on the targeted sets to focus on specific subject areas.
Nick used each set to train active learning’s artificial intelligence functionality to find hot documents in subsequent sets that the white-collar team would then analyze. Running parallel active learning projects allowed Foley to have coverage over the entire data set to find responsive documents. After the likely responsive documents were found, Foley’s contract attorneys took advantage of active learning’s prioritized review queue in the more distilled data set to surface documents likely to be responsive to the front of the review queue.
The workflow identified three million records that were likely to be responsive from the main 117 million document data set. Of those three million, the prioritized review bubbled up 23,000 documents for immediate review and production. For each of the random sample indexes, active learning found 6,000 responsive documents in each set that didn’t contain any search terms—including a document that Foley & Lardner’s contract reviewers had originally marked as non-responsive because it didn’t match any of the search terms. This document laid out the key players and their activities, which were the focus of the investigation.
These results convinced the firm’s white-collar attorneys that active learning was the best option to find the additional responsive documents.
All in all, RelativityOne and active learning saved Foley & Lardner an estimated $10 million dollars and allowed them to complete the investigation 10 times faster.
Moreover, RelativityOne allowed Foley & Lardner to execute on this case with ease and accuracy. For instance, the firm’s Relativity Server instance could only support up to five million documents. Without RelativityOne, they would have had to outsource the case, which would have been a huge expense.
As this was the largest case hosted in RelativityOne, Nick cited the Relativity team as a core reason that this case went smoothly. Nick’s team worked closely with Relativity’s support and customer success management teams to ensure the workflows and technology worked hand in hand.
“We would not have been able to scale this case on our own. Thanks to the diligent and effcient team members who worked with us, we were able to conduct this investigation smoothly,” Nick said.
“Everyone from Foley working on this case is happy with how active learning is going. We are going to use this as an example to help evangelize using Relativity Analytics with other case teams.”
NICK COLE, Director of Litigation Support
Want to learn more about RelativityOne?