SPLADE: Sparse Lexical and Expansion Model for First Stage Ranking
arXiv:2107.05720
Abstract
In neural Information Retrieval, ongoing research is directed towards improving the first retriever in ranking pipelines. Learning dense embeddings to conduct retrieval using efficient approximate nearest neighbors methods has proven to work well. Meanwhile, there has been a growing interest in learning sparse representations for documents and queries, that could inherit from the desirable properties of bag-of-words models such as the exact matching of terms and the efficiency of inverted indexes. In this work, we present a new first-stage ranker based on explicit sparsity regularization and a log-saturation effect on term weights, leading to highly sparse representations and competitive results with respect to state-of-the-art dense and sparse methods. Our approach is simple, trained end-to-end in a single stage. We also explore the trade-off between effectiveness and efficiency, by controlling the contribution of the sparsity regularization.
arXiv comment5 pages, SIGIR'21 short paper
References in corpus (11)
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- HuggingFace's Transformers: State-of-the-art Natural Language Processing
- REALM: Retrieval-Augmented Language Model Pre-Training
- Passage Re-ranking with BERT
- Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval
- Document Expansion by Query Prediction
- Context-Aware Sentence/Passage Term Importance Estimation For First Stage Retrieval
- RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answering
- Overview of the TREC 2019 deep learning track
- Distilling Dense Representations for Ranking using Tightly-Coupled Teachers
- SPARTA: Efficient Open-Domain Question Answering via Sparse Transformer Matching Retrieval