Corporate clients are losing patience for law firms who fail to innovate in a competitive marketplace: a survey conducted by Thompson Hine indicates that 81 percent of clients’ in-house counsel and senior executives see “little” to “hardly any” innovation in their law firms. The survey also uncovered desired areas of improvement including reduced costs, improved project management, better staffing, and technology. On the plus side, this means great opportunity for improvement in the litigation process—especially document review, which accounts for more than 70 percent of all e-discovery costs.
While there have been significant improvements to the tooling and operational portions of e-discovery, little investment has been made in the most chaotic, dynamic component of the review process: the human element. The technology available to management and reviewers is mostly results-driven, with less attention paid to the behavior that generate those results. Now, with the growth of virtual teams in document review, the need for real-time insights to drive higher work quality and productivity is greater than ever before.
The Current Method of Analyzing of Reviewer Performance is Broken
In legal document review, measuring true reviewer performance is challenging but undeniably important; their efficiency is critical to successful e-discovery. Reviewers must be productive, use tools effectively, review accurately, and communicate well to maximize their efficiency. However, there are several problems with many typical methods for analyzing reviewer performance today.
Challenges to Understanding True Reviewer Performance
- Analytics are pulled manually and referenced without the context of work time performance and behavior.
- Most case managers are looking at data points associated with reviewer performance only once a day or less.
- Reviewers have traditionally been measured using one-dimensional metrics.
- Metrics used to measure document reviewer costs are examined in retrospect and are output-centric, focusing primarily on review costs per document.
In a world where an increasing number of people use wearable technology to monitor their physical activity with the goal of self-improvement, it seems obvious that an industry plagued by excessive costs and operational inefficiencies would benefit from similarly innovative tools that provide a real-time understanding of document reviewer performance. Doing so can help legal teams tackle productivity and performance challenges to better deliver on their clients’ needs.
Applying Real-Time Analytics Drastically Improves Productivity
Traditional reviewer productivity measurement usually includes a spreadsheet with metrics like document review count (the number of documents that reviewer made a coding decision on in a specific period) and overturns (the number of documents for which their coding decision was overruled by another person or process)—both of which are focused on output and do not reflect the complete picture of a reviewer’s day. Generally, these are evaluated after work activity is complete, rather than during, which limits the ability for real-time performance adjustment.
Applying Real-time Data to Evaluate Reviewer Performance
Reviewers’ days consist of much more than just document review. In fact, most reviewers use an average of 7-10 work-related applications or web tools/services as part of their day-to-day work, spend hours a week in meetings, and take short breaks throughout the day. It’s also no secret that as humans, we get distracted sometimes. Distractions can result in hours per week of lost productivity, spent on news sites, social media, texting, chatting, music, and more. And with cell phones at their disposal for most the day, employees can also easily access websites and applications blocked by company IT departments.
In terms of work time, an average day in the life of a document reviewer may look something like this:
With a better understanding of how a reviewer spends their day, opportunities for discussion and potential improvement are clear. Using the example above, a manager could identify that this reviewer is spending a decent amount of time in non-work sites or applications, they may be reviewing documents in a batch that is Excel-heavy, or are spending inefficient time in their work intranet.
Access to this data in real-time means a reviewer or case manager can evaluate and make adjustments as needed immediately—whether that’s by changing up batches so the reviewer is more engaged, or ensuring the right reference material is easy to find in the team’s intranet. Pairing valuable work time analytics with traditional case metrics tells a much more colorful story about reviewer productivity and corresponding performance.
Even with this basic example, it’s clear that spreadsheets containing a few high-level metrics often don’t provide important context that could have a significant impact on implementing improvements in reviewer productivity. A more real-time view of performance can result in marked workforce improvements for review teams. For reviewers, visibility into their own real-time activity can help reduce the temptation for distraction and increase individual accountability to stay on task. Additionally, reviewers can use real-time data to analyze and self-manage their performance.
Making Improvements Across Teams
For case managers seeking to make more sweeping productivity improvements, the benefits are similar, but allow for scalable management applied across a team of reviewers. With a large and/or remote review team, managing performance can be daunting and nearly impossible in real time without the right data. Better access to reviewer activity and performance data can help managers:
- Identify efficient work habits of top reviewers and apply those learnings to reviewer training.
- Increase accountability and personal performance management tools, driving good reviewers to become great.
- Remove document review roadblocks by identifying and addressing work inefficiencies.
- Coach reviewers to improvement.
There’s also additional strategic value for litigation executives. Because clients are eager for innovation and change, the application of real-time analytics helps firms not only deliver on those objectives, but offer more transparency and, sometimes, justification around billable hours or alternative fee structures as value adds. This value will become increasingly important as regulation and cultural norms around pricing models continue to evolve with the legal landscape.
Real-Life Use Case: Using Real-Time Analytics to Improve Productivity
At Esquify, as part of our platform, reviewer work time is detailed for both the reviewer and case manager to see, allowing for increased accountability, self-managed performance, and actionable insights for management. Combining real-time work time analytics with traditional document review metrics is a key part of our strategy in a competitive marketplace, offering a more comprehensive view of performance versus standard e-discovery reporting. To highlight the effectiveness of real-time analytics with a simple use case, let’s walk through the real-life experience of an Esquify customer.
In one of our customer’s recent projects, a reviewer was lagging consistently behind in terms of documents reviewed per day. With only that single data point to reference, the data tells a limited story. To get to the heart of it, a review manager must try and either press the reviewer for details—otherwise, the reviewer is considered to be a poor performer and ultimately churns.
With access to real-time work and productivity analytics in Esquify, the case manager was able to identify that over 20 percent of the reviewer’s time in a day was spent in MS Word. The case manager quickly engaged the reviewer to understand why so much of their work time was spent in that application. As it turns out, the reviewer was documenting his coding decisions to make sure he can justify his choice later if questioned about it during the QC or Level 2 review. A quick conversation to address the perceived need for, and then adjust, this behavior occurred between the case manager and reviewer, and an issue that could’ve dragged on unrecognized—or led to a premature dismissal—was easily resolved.
As this simple use case demonstrates, reviewer performance comes down to more than just a few metrics in spreadsheets. The primary metric used to measure their performance is the number of documents they complete—but they do more than just review documents as part of their normal course of work, and those tasks have a direct impact on their resulting productivity, too. Reviewers participate in internal meetings, engage with many work and non-work related tools, and communicate internally with their team members. Case managers and ultimately their firms can win big if they’re able to optimize team efficiency and productivity at scale.
