MS MARCO: Benchmarking Ranking Models in the Large-Data Regime
arXiv:2105.04021
Abstract
Evaluation efforts such as TREC, CLEF, NTCIR and FIRE, alongside public leaderboard such as MS MARCO, are intended to encourage research and track our progress, addressing big questions in our field. However, the goal is not simply to identify which run is "best", achieving the top score. The goal is to move the field forward by developing new robust techniques, that work in many different settings, and are adopted in research and practice. This paper uses the MS MARCO and TREC Deep Learning Track as our case study, comparing it to the case of TREC ad hoc ranking in the 1990s. We show how the design of the evaluation effort can encourage or discourage certain outcomes, and raising questions about internal and external validity of results. We provide some analysis of certain pitfalls, and a statement of best practices for avoiding such pitfalls. We summarize the progress of the effort so far, and describe our desired end state of "robust usefulness", along with steps that might be required to get us there.
References in corpus (15)
- CORD-19: The COVID-19 Open Research Dataset
- End-to-End Neural Ad-hoc Ranking with Kernel Pooling
- Passage Re-ranking with BERT
- Query Resolution for Conversational Search with Limited Supervision
- Pretrained Transformers for Text Ranking: BERT and Beyond
- Overview of the TREC 2019 deep learning track
- Overview of the TREC 2021 deep learning track
- Rapidly Deploying a Neural Search Engine for the COVID-19 Open Research Dataset: Preliminary Thoughts and Lessons Learned
- TREC CAsT 2019: The Conversational Assistance Track Overview
- TREC-COVID: Constructing a Pandemic Information Retrieval Test Collection
- Document Ranking with a Pretrained Sequence-to-Sequence Model
- RRF102: Meeting the TREC-COVID Challenge with a 100+ Runs Ensemble
- Benchmark for Complex Answer Retrieval
- CMT in TREC-COVID Round 2: Mitigating the Generalization Gaps from Web to Special Domain Search
- Long Document Ranking with Query-Directed Sparse Transformer