SpaDE: Improving Sparse Representations using a Dual Document Encoder for First-stage Retrieval
arXiv:2209.05917 · doi:10.1145/3511808.3557456
Abstract
Sparse document representations have been widely used to retrieve relevant documents via exact lexical matching. Owing to the pre-computed inverted index, it supports fast ad-hoc search but incurs the vocabulary mismatch problem. Although recent neural ranking models using pre-trained language models can address this problem, they usually require expensive query inference costs, implying the trade-off between effectiveness and efficiency. Tackling the trade-off, we propose a novel uni-encoder ranking model, Sparse retriever using a Dual document Encoder (SpaDE), learning document representation via the dual encoder. Each encoder plays a central role in (i) adjusting the importance of terms to improve lexical matching and (ii) expanding additional terms to support semantic matching. Furthermore, our co-training strategy trains the dual encoder effectively and avoids unnecessary intervention in training each other. Experimental results on several benchmarks show that SpaDE outperforms existing uni-encoder ranking models.
In Proceedings of the 31st ACM International Conference on Information and Knowledge Management (CIKM '22). 13 pages
References in corpus (14)
- A Deep Relevance Matching Model for Ad-hoc Retrieval
- End-to-End Neural Ad-hoc Ranking with Kernel Pooling
- Deeper Text Understanding for IR with Contextual Neural Language Modeling
- Multi-Stage Document Ranking with BERT
- Context-Aware Sentence/Passage Term Importance Estimation For First Stage Retrieval
- Overview of the TREC 2020 deep learning track
- A Few Brief Notes on DeepImpact, COIL, and a Conceptual Framework for Information Retrieval Techniques
- SparTerm: Learning Term-based Sparse Representation for Fast Text Retrieval
- Unsupervised Corpus Aware Language Model Pre-training for Dense Passage Retrieval
- Fast Passage Re-ranking with Contextualized Exact Term Matching and Efficient Passage Expansion
- Wacky Weights in Learned Sparse Representations and the Revenge of Score-at-a-Time Query Evaluation
- Learning Discrete Representations via Constrained Clustering for Effective and Efficient Dense Retrieval
- Sparsifying Sparse Representations for Passage Retrieval by Top- Masking
- SPLADE: Sparse Lexical and Expansion Model for First Stage Ranking