by Constantine Pappas on July 25, 2012
How to Deal with Non-Responsive Documents that Contain Responsive Language
After posting a couple months back about our Reviewer Protocol document—which outlines best practices for identifying good example documents while using computer-assisted review—we received a number of requests for it. Though we realize it’s not the Magna Carta or Detective Comics #27, we hope it can be a helpful reference for anyone conducting a Relativity Assisted Review project. We thought it could be useful to dive into some of its content more deeply, and share some detail on how collaborating with our users has helped us continue to improve the protocol.
Several months ago, we conducted a Relativity Assisted Review training session at a client’s office. (To set the stage, the virtual ink was not yet dry on our protocol document.) As we were going through the document with the attendees, a young attorney asked a question: “Excuse me, but what about early drafts of contracts? If we’ve agreed to carve those out and mark them as non-responsive, won’t that teach the system incorrectly?”
This was an incredibly practical observation that can only come from out in the wild, with the extra end-user perspective that drives this kind of collaboration. How should you handle a document that has responsive language but should be marked as non-responsive for external reasons?
There are two things to keep in mind.
1. The fact that it’s a carve-out makes it technically non-responsive, even if it’s comprised of highly responsive language.
2. Best practice should be to mark this draft as non-responsive and then exclude it as an example document, so the computer won’t be thrown off by the content.
After receiving this insightful suggestion, we went back and added a section to the Reviewer Protocol document called “False Negatives.” The section is meant to be a helpful guide for documents which, through an agreement between both parties, are cut out as non-responsive for more complex reasons (like subject matter, creation date, or point of origin).
Perhaps we didn’t invent the concept of monarchal accountability or introduce the world to Batman, but we did learn something from the field that we were able to share with others. As always, if you’d like to request a copy of the Reviewer Protocol document—or have any questions or suggestions for it—please feel free to email us at firstname.lastname@example.org.