Your single source for new lessons on legal technology, e-discovery, and the people innovating behind the scenes.

Business Intelligence: Tracking Metrics Across Projects

This blog post is a thorough introduction to the ways high-level metrics from your e-discovery projects, when viewed holistically, can help influence a smarter e-discovery strategy for your business. It was originally published on the Advanced Discovery blog as part of their recent series on business intelligence.

In the first part of this short blog series, we began by reviewing core business intelligence concepts and how they can apply during an individual e-discovery project. Here, we review how they apply across individual e-discovery projects as part of effective e-discovery program management.

Moving Beyond the Individual Project

As we noted, each individual e-discovery project provides opportunities for tracking and aggregating useful metrics about both substantive elements of the project (i.e., sources, materials, and their contents) and formal elements of the project (i.e., tools, individuals, and their processes). Tracking these metrics can provide insights that lead to improvements in efficiency and efficacy during a project, but there is third category of macro elements about which we can also track metrics to facilitate overall program management.

Macro elements are those that are unlikely to have utility during an individual project but that, if tracked, can still lead to improvements across projects. For example, the following are all macro elements about which metrics might be tracked to aid in program management:

  • Pre-discovery Negotiation Efforts and Their Effects
  • Internal/External Resource Selection and Allocation
  • Relative Efficacy of Different ECA and Review Strategies
  • Costs, Broken Down by Resource, Process, and More
  • Duration and Throughput for Resources and Processes
  • Overlaps in Matters’ Scope, Implicated Sources, Etc.

Beyond these specific examples, tracking key metrics across projects, in an organized, aggregated way, makes it possible to establish efficiency, efficacy, and cost benchmarks for your processes and activities. And, from those benchmarks, concrete goals for iterative improvement can then be set.

Macro Metrics and Their Program Management Applications

Next, let’s review some of the major subcategories of macro, cross-project metrics that you might track and examples of the benefits that tracking them can yield.

Data Metrics

This subcategory of metrics covers information about your data and where it comes from, including: sources; source types; individual custodians; departmental custodians; mobile device sources; cloud sources; social media sources; file types; and, volumes for all of these and more. Tracking this data across projects can yield a variety of potential benefits:

  • More accurately predict your collection resource needs for projects and the future
  • Identify overlapping or repeated collections that can be combined or streamlined
  • Compare relative cost/efficiency of various tools, techniques, and service providers

Processing Metrics

This subcategory of metrics covers information about your processing activities, including: platforms and tools used; throughput and error rates achieved; variations in completion time for various source types; filtering techniques employed and their rates of volume reduction; and, hours of employee work required per job and more. Tracking this data across projects can also yield a variety of potential benefits:

  • Evaluate the relative performance of different tools on speed and accuracy
  • Better predict the time and resources required for any given processing job
  • Measure the efficacy of objective filtering techniques for various purposes

ECA Metrics

This subcategory of metrics covers information about your early case assessment activities, including: platforms and tools used; search and analytics features applied; volume reductions achieved; time spent; and, reviewers required and more. Tracking this data across projects can also yield a variety of potential benefits:

  • Learn what approaches work best for what source or case types
  • Compare ECA data volume reductions with review results to benchmark ECA
  • Analyze assorted analytic tools for relative time required and benefit achieved

Review Metrics

Within a single e-discovery project, review metrics are invaluable for team management and oversight, but across projects additional variables can be tracked, including: platforms and tools used; batch sizes and batch organization used; review team sizes and team organization used; application of email threading and near-duplicate grouping; privilege logging and redaction workflows employed; and, details of any technology-assisted review (TAR) process employed. Tracking this data across projects can also yield a variety of potential benefits:

  • Measure the comparative efficiency of various team structures and workflows
  • Discover optimal batch size and document organization strategies for your reviews
  • Answer for your organization the question of whether and when TAR is worth it

Cross-Phase Metrics

This subcategory of metrics covers additional useful information that cuts across several or all phases of an e-discovery project, including: matter type; jurisdiction; value at risk; outcome achieved; total project costs and total costs per document ultimately produced; internal resources and outside service providers employed; and, overall ratio of documents collected to documents produced and more. Tracking this additional data across projects is invaluable for overall program management and can yield a variety of potential benefits:

  • Spot trends in your litigation portfolio that can aid in program planning and budgeting
  • Establish overall program benchmarks for cost-per-document, collection efficiency, etc.
  • Determine which tasks and phases are best handled internally and which externally

As you can see from this list of examples, the range of things you might track project-to-project is vast, and the range of insights you might gain is equally so. A key question for each organization will be which metrics are worth the cost and trouble of tracking, and that will be dictated, in part, by your long-term goals for the effort.