On Minimizing Cost in Legal Document Review Workflows
arXiv:2106.09866 · doi:10.1145/3469096.3469872
Abstract
Technology-assisted review (TAR) refers to human-in-the-loop machine learning workflows for document review in legal discovery and other high recall review tasks. Attorneys and legal technologists have debated whether review should be a single iterative process (one-phase TAR workflows) or whether model training and review should be separate (two-phase TAR workflows), with implications for the choice of active learning algorithm. The relative cost of manual labeling for different purposes (training vs. review) and of different documents (positive vs. negative examples) is a key and neglected factor in this debate. Using a novel cost dynamics analysis, we show analytically and empirically that these relative costs strongly impact whether a one-phase or two-phase workflow minimizes cost. We also show how category prevalence, classification task difficulty, and collection size impact the optimal choice not only of workflow type, but of active learning method and stopping point.
10 pages, 3 figures. Accepted at DocEng 21
Cited by in corpus (5)
- Heuristic Stopping Rules For Technology-Assisted Review
- Certifying One-Phase Technology-Assisted Reviews
- Stopping Methods for Technology Assisted Reviews based on Point Processes
- TARexp: A Python Framework for Technology-Assisted Review Experiments
- Contextualization with SPLADE for High Recall Retrieval