SparTerm: Learning Term-based Sparse Representation for Fast Text Retrieval

Abstract

Term-based sparse representations dominate the first-stage text retrieval in industrial applications, due to its advantage in efficiency, interpretability, and exact term matching. In this paper, we study the problem of transferring the deep knowledge of the pre-trained language model (PLM) to Term-based Sparse representations, aiming to improve the representation capacity of bag-of-words(BoW) method for semantic-level matching, while still keeping its advantages. Specifically, we propose a novel framework SparTerm to directly learn sparse text representations in the full vocabulary space. The proposed SparTerm comprises an importance predictor to predict the importance for each term in the vocabulary, and a gating controller to control the term activation. These two modules cooperatively ensure the sparsity and flexibility of the final text representation, which unifies the term-weighting and expansion in the same framework. Evaluated on MSMARCO dataset, SparTerm significantly outperforms traditional sparse methods and achieves state of the art ranking performance among all the PLM-based sparse models.

References in corpus (3)

• Document Expansion by Query Prediction
• Context-Aware Sentence/Passage Term Importance Estimation For First Stage Retrieval
• Sparse, Dense, and Attentional Representations for Text Retrieval

Cited by in corpus (13)

• Information Retrieval: Recent Advances and Beyond
• Semantic Models for the First-stage Retrieval: A Comprehensive Review
• Pretrained Transformers for Text Ranking: BERT and Beyond
• Wacky Weights in Learned Sparse Representations and the Revenge of Score-at-a-Time Query Evaluation
• To Interpolate or not to Interpolate: PRF, Dense and Sparse Retrievers
• Optimizing Guided Traversal for Fast Learned Sparse Retrieval
• SpaDE: Improving Sparse Representations using a Dual Document Encoder for First-stage Retrieval
• SLIM: Sparsified Late Interaction for Multi-Vector Retrieval with Inverted Indexes
• Learning Passage Impacts for Inverted Indexes
• An Analysis on Matching Mechanisms and Token Pruning for Late-interaction Models
• Exploring ${\ell }_0$ Sparsification for Inference-free Sparse Retrievers
• Ultra-High Dimensional Sparse Representations with Binarization for Efficient Text Retrieval
• Searching Personal Collections