As the use of Assisted Review is becoming more mainstream, we’re seeing more clients beginning to use it. When they get started, we hear some consistent questions about the workflow and how to best approach each project.
To help you take a smarter approach to your Assisted Review projects, the following are straightforward answers and insights for a few common questions.
Are control sets necessary?
Control sets are not necessary to execute a project. The control set provides information that might be helpful in determining project progress, but there are other options and many reasons to go with a different approach. Control sets provide a measure of precision and recall—two statistics than can help measure your project’s progress and success. If you don’t need these measurements and feel comfortable with measuring success based on overturns, then a control set isn’t necessary.
The idea behind a control set is to set aside a segment of documents that represent the total population. If your population changes, your control set should change too—so adding or removing documents from the population should spark an update to your control set. If you know at the start of a project that your data will change, a control set might not be a great first round.
How many training rounds are necessary?
Training rounds are where a project starts—they help teach Relativity about your data. There isn’t an exact number of rounds to follow. The goal to get past training is to get a categorized value for as many documents as possible, so the amount and content of the documents will change the number of items necessary for training. After the document set has a large percentage of items categorized, you can begin the QC process and continue iteratively. Keep in mind that, if new documents are added, you may need to perform additional training rounds.
What does a quality control round accomplish?
QC rounds are the most important part of the process. They do two things: train the system on reviewers’ decisions for new document values and, most importantly, allow you to measure the error rate of the machine. QC rounds should always include at least a statistically valid sample size. That number can be calculated right inside Relativity, and following it helps ensure that the updates you’ve implemented to the system can be measured to help you understand how the project should proceed and, most importantly, when it is complete.
What makes an efficient project?
An educated reviewer is the number one factor in making a project successful. The consistency of their coding decisions—and a comprehensive understanding of how training the system works—is very important in a reviewer. Setting expectations upfront is also very valuable. Running an Assisted Review project takes a team, and a successful team needs coaching. We recommend that the project administrator take on this role to ensure the reviewers understand their tasks. An Assisted Review project isn’t quite the same as a traditional review. Your team should use the tool to leverage their expertise but, to be effective, they’ll need to do it the right way.
I hope these thoughts will help with get you started with your Assisted Review project. As always, if you’re starting a project and could use a second set of eyes to help you evaluate your strategy, you can always reach out to the advice@kCura team and let them guide you on the right path.
