Heuristic Stopping Rules For Technology-Assisted Review

Abstract

Technology-assisted review (TAR) refers to human-in-the-loop active learning workflows for finding relevant documents in large collections. These workflows often must meet a target for the proportion of relevant documents found (i.e. recall) while also holding down costs. A variety of heuristic stopping rules have been suggested for striking this tradeoff in particular settings, but none have been tested against a range of recall targets and tasks. We propose two new heuristic stopping rules, Quant and QuantCI based on model-based estimation techniques from survey research. We compare them against a range of proposed heuristics and find they are accurate at hitting a range of recall targets while substantially reducing review costs.

arXiv comment10 pages, 2 figures. Accepted at DocEng 21

References in corpus (1)

• On Minimizing Cost in Legal Document Review Workflows

Cited by in corpus (6)

• Certifying One-Phase Technology-Assisted Reviews
• Stopping Methods for Technology Assisted Reviews based on Point Processes
• TARexp: A Python Framework for Technology-Assisted Review Experiments
• Dense Retrieval with Continuous Explicit Feedback for Systematic Review Screening Prioritisation
• RLStop: A Reinforcement Learning Stopping Method for TAR
• Contextualization with SPLADE for High Recall Retrieval