How did they do it?
- Used Assisted Review to identify responsive documents
- In total, reviewed just 2.5 percent of the original 1.8 million documents
Legility client Thompson & Knight LLP, a full-service law firm with 11 offices worldwide, was working on an internal investigation for a client. Faced with a year-end audit deadline for the client, Thompson & Knight had an immediate need to review 1.8 million records for their response. Even after they performed keyword culling, tens of thousands of documents remained to be reviewed.
Though the Thompson & Knight team wasn’t expecting to produce any of the data, they needed to get their hands on the most relevant documents as soon as possible, so the investigation could be completed.
Danny Thankachan, a litigation support manager and e-discovery consultant at Thompson & Knight, said the project was unprecedented at the firm because of its short timeframe. His team turned to partner Legility to lead them through the fast-paced situation.
“We knew Legility had the ability and confidence to handle this, and to make it easy for my team,” said Danny.
Thompson & Knight were confident that, when they handed Legility their records on Friday, the Legility team would be able to help them find the most relevant documents to start digging into by Monday morning. They were also confident that using computer-assisted review was their best bet for tackling the project.
The goal was to send the most relevant documents and related data to reviewers as soon as possible, helping the case team learn as much as possible about the data set—and the matter at hand—early on.
Initially, Legility had doubts about how quickly the team could get through so many documents.
“We pushed back on the timeline at first. It’s hard to tell so early on if you have the kind of data you need to run a computer-assisted review project so quickly. In the end, though, we set our sights on Thompson & Knight’s end-of-weekend deadline and stepped up to the challenge.”
“We set our sights on Thompson & Knight’s end-of-weekend deadline and stepped up to the challenge.”
The team decided to use Assisted Review. Thompson & Knight is skilled with the platform and were seeking the help of Inventus to leverage Relativity as effectively as possible on such a high-stakes challenge.
“The integration across Relativity is seamless, and that’s really helpful. We process once, run analytics, and review all the data in a single environment, so it was easy to get started and jump right in.”
Before Legility started the process, they called Relativity’s solution specialists for validation on their project plan.
The Legility team called Relativity’s team to confirm that this could be done. We wanted to know if we were trying to fit a square peg in a round hole. They gave us the reassurance that we needed and provided support throughout as the project progressed.
Thompson & Knight ran search terms against the initial 1.8 million records, narrowing down the set to 86,000 documents on Friday night. That provided Legility with a more manageable, more targeted data set they could analyze with Assisted Review. On Saturday morning, they began the project. To make the most of their initial training of the system, three expert reviewers were put on the case—two associates from Thompson & Knight and a financial expert, all of whom were knowledgeable about the project.
Legility instructed the reviewers to practice “coding by consensus,” meaning they would come to collective agreements on responsiveness and code the documents according to those standards. They sat in a room together to work through 1,500 initial training documents and held discussions whenever a debatable document came their way. This meant each training decision was a group effort that resulted in consistent coding across the board, enabling the system to learn more quickly and effectively—and preventing future downtime spent resolving incorrect coding decisions that would’ve set the software on the wrong path.
After Assisted Review came back with the first set of coding decisions based on the training documents, Legility began their first quality control round to ensure the system would be on the right track in coding the rest of the document universe. During that round, the same reviewers were served a random sample of documents coded by Relativity. In the end, they disagreed with only 10 percent of the decisions—an incredibly low overturn rate at such an early stage. The Legility case team were happy with those results.
Still, because they were pleasantly surprised by the quick completion, the Legility team performed an additional QC round “just to make sure those results weren’t a fluke.” In this second round, the overturn rate decreased to 9 percent, which was enough proof for the team to confidently rely on the computer’s accuracy in prioritizing the most relevant documents. When Legility finished the Assisted Review project on Sunday night, the team was confident in the results they were giving to Thompson & Knight the next morning.
On Monday morning, with their documents in hand— prioritized first by what was coded as responsive and second by what was uncategorized—Thompson & Knight were ready to move forward with contract review. The responsive set they brought to attorneys made up just 12 percent of the 86,000 documents analyzed by Assisted Review.
“Assisted Review helped us quickly identify relevant documents and categorize hot documents that needed immediate attention. It provided immense value in an incredibly short period of time.”
By the time they finished the initial Assisted Review project, Legility’s project team had manually reviewed only 2,100 documents—a result that both Legility and Thompson & Knight found to be “astounding.” Even after Thompson & Knight’s team manually reviewed the responsive and uncategorized data, having the documents prioritized meant both teams ended up reviewing only 2.5 percent of the original 1.8 million documents in total.
“The process went really smoothly,” said Danny. “Assisted Review helped us quickly identify relevant documents and categorize hot documents that needed immediate attention. It provided immense value in an incredibly short period of time.”
|Key Project Stats
|Total documents to be reviewed for their response
|Number of days to identify potentially responsive documents
|Total documents considered for Assisted Review
|Overturn rate by second QC round
|Total documents manually reviewed for training and QC
|Percent of documents analyzed by Assisted Review that were responsive
|Total documents in the responsive set
|Percent of total documents manually reviewed by both teams