Let's measure run time! Extending the IR replicability infrastructure to include performance aspects
arXiv:1907.04614
Abstract
Establishing a docker-based replicability infrastructure offers the community a great opportunity: measuring the run time of information retrieval systems. The time required to present query results to a user is paramount to the users satisfaction. Recent advances in neural IR re-ranking models put the issue of query latency at the forefront. They bring a complex trade-off between performance and effectiveness based on a myriad of factors: the choice of encoding model, network architecture, hardware acceleration and many others. The best performing models (currently using the BERT transformer model) run orders of magnitude more slowly than simpler architectures. We aim to broaden the focus of the neural IR community to include performance considerations -- to sustain the practical applicability of our innovations. In this position paper we supply our argument with a case study exploring the performance of different neural re-ranking models. Finally, we propose to extend the OSIRRC docker-based replicability infrastructure with two performance focused benchmark scenarios.
arXiv commentPosition paper @ SIGIR 2019 Open-Source IR Replicability Challenge (OSIRRC)
References in corpus (6)
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
- End-to-End Neural Ad-hoc Ranking with Kernel Pooling
- Passage Re-ranking with BERT
- Document Expansion by Query Prediction
- Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned
- An Updated Duet Model for Passage Re-ranking
Cited by in corpus (6)
- ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT
- Semantic Models for the First-stage Retrieval: A Comprehensive Review
- Pretrained Transformers for Text Ranking: BERT and Beyond
- Multistage BiCross encoder for multilingual access to COVID-19 health information
- An In-depth Analysis of Passage-Level Label Transfer for Contextual Document Ranking
- Mitigating the Position Bias of Transformer Models in Passage Re-Ranking