Stopping Methods for Technology Assisted Reviews based on Point Processes

Abstract

Technology Assisted Review (TAR), which aims to reduce the effort required to screen collections of documents for relevance, is used to develop systematic reviews of medical evidence and identify documents that must be disclosed in response to legal proceedings. Stopping methods are algorithms which determine when to stop screening documents during the TAR process, helping to ensure that workload is minimised while still achieving a high level of recall. This paper proposes a novel stopping method based on point processes, which are statistical models that can be used to represent the occurrence of random events. The approach uses rate functions to model the occurrence of relevant documents in the ranking and compares four candidates, including one that has not previously been used for this purpose (hyperbolic). Evaluation is carried out using standard datasets (CLEF e-Health, TREC Total Recall, TREC Legal), and this work is the first to explore stopping method robustness by reporting performance on a range of rankings of varying effectiveness. Results show that the proposed method achieves the desired level of recall without requiring an excessive number of documents to be examined in the majority of cases and also compares well against multiple alternative approaches.

arXiv commentAccepted by ACM Transactions on Information Systems (TOIS)

References in corpus (10)

• FAST${}^2$: an Intelligent Assistant for Finding Relevant Papers
• Autonomy and Reliability of Continuous Active Learning for Technology-Assisted Review
• Active Sampling for Large-scale Information Retrieval Evaluation
• Neural Rankers for Effective Screening Prioritisation in Medical Systematic Review Literature Search
• Heuristic Stopping Rules For Technology-Assisted Review
• On Minimizing Cost in Legal Document Review Workflows
• Certifying One-Phase Technology-Assisted Reviews
• TARexp: A Python Framework for Technology-Assisted Review Experiments
• Goldilocks: Just-Right Tuning of BERT for Technology-Assisted Review
• Providing More Efficient Access To Government Records: A Use Case Involving Application of Machine Learning to Improve FOIA Review for the Deliberative Process Privilege

Cited by in corpus (5)

• LLM-Assisted Relevance Assessments: When Should We Ask LLMs for Help?
• RLStop: A Reinforcement Learning Stopping Method for TAR
• The Next Phase of Scientific Fact-Checking: Advanced Evidence Retrieval from Complex Structured Academic Papers
• A Generalised and Adaptable Reinforcement Learning Stopping Method
• Contextualization with SPLADE for High Recall Retrieval