Overview of the TREC 2021 deep learning track

Abstract

This is the third year of the TREC Deep Learning track. As in previous years, we leverage the MS MARCO datasets that made hundreds of thousands of human annotated training labels available for both passage and document ranking tasks. In addition, this year we refreshed both the document and the passage collections which also led to a nearly four times increase in the document collection size and nearly $16$ times increase in the size of the passage collection. Deep neural ranking models that employ large scale pretraininig continued to outperform traditional retrieval methods this year. We also found that single stage retrieval can achieve good performance on both tasks although they still do not perform at par with multistage retrieval pipelines. Finally, the increase in the collection size and the general data refresh raised some questions about completeness of NIST judgments and the quality of the training labels that were mapped to the new collections from the old ones which we discuss in this report.

Cited by in corpus (36)

• Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval
• Pretrained Transformers for Text Ranking: BERT and Beyond
• Sparse, Dense, and Attentional Representations for Text Retrieval
• Complementing Lexical Retrieval with Semantic Residual Embedding
• Distilling Dense Representations for Ranking using Tightly-Coupled Teachers
• Fast Passage Re-ranking with Contextualized Exact Term Matching and Efficient Passage Expansion
• Neural Passage Retrieval with Improved Negative Contrast
• mMARCO: A Multilingual Version of the MS MARCO Passage Ranking Dataset
• Conformer-Kernel with Query Term Independence for Document Retrieval
• Leveraging Semantic and Lexical Matching to Improve the Recall of Document Retrieval Systems: A Hybrid Approach
• Learning To Retrieve: How to Train a Dense Retrieval Model Effectively and Efficiently
• Deep Reinforced Query Reformulation for Information Retrieval
• SPLADE v2: Sparse Lexical and Expansion Model for Information Retrieval
• Optimizing Dense Retrieval Model Training with Hard Negatives
• Intra-Document Cascading: Learning to Select Passages for Neural Document Ranking
• CMT in TREC-COVID Round 2: Mitigating the Generalization Gaps from Web to Special Domain Search
• Learning Passage Impacts for Inverted Indexes
• Simplified TinyBERT: Knowledge Distillation for Document Retrieval
• On Single and Multiple Representations in Dense Passage Retrieval
• Improving Document Representations by Generating Pseudo Query Embeddings for Dense Retrieval
• Neural document expansion for ad-hoc information retrieval
• Pseudo Relevance Feedback with Deep Language Models and Dense Retrievers: Successes and Pitfalls
• Interpreting Dense Retrieval as Mixture of Topics
• Modeling Relevance Ranking under the Pre-training and Fine-tuning Paradigm
• Improving Query Representations for Dense Retrieval with Pseudo Relevance Feedback
• Jointly Optimizing Query Encoder and Product Quantization to Improve Retrieval Performance
• Query2Prod2Vec Grounded Word Embeddings for eCommerce
• Few-Shot Text Ranking with Meta Adapted Synthetic Weak Supervision
• TREC Deep Learning Track: Reusable Test Collections in the Large Data Regime
• Modularized Transfomer-based Ranking Framework
• Mitigating the Position Bias of Transformer Models in Passage Re-Ranking
• Explaining Documents' Relevance to Search Queries
• Contrastive Document Representation Learning with Graph Attention Networks
• Extreme Multi-label Learning for Semantic Matching in Product Search
• MS MARCO: Benchmarking Ranking Models in the Large-Data Regime
• Less is More: Pre-train a Strong Text Encoder for Dense Retrieval Using a Weak Decoder