From Distillation to Hard Negative Sampling: Making Sparse Neural IR Models More Effective
arXiv:2205.04733
Abstract
Neural retrievers based on dense representations combined with Approximate Nearest Neighbors search have recently received a lot of attention, owing their success to distillation and/or better sampling of examples for training -- while still relying on the same backbone architecture. In the meantime, sparse representation learning fueled by traditional inverted indexing techniques has seen a growing interest, inheriting from desirable IR priors such as explicit lexical matching. While some architectural variants have been proposed, a lesser effort has been put in the training of such models. In this work, we build on SPLADE -- a sparse expansion-based retriever -- and show to which extent it is able to benefit from the same training improvements as dense models, by studying the effect of distillation, hard-negative mining as well as the Pre-trained Language Model initialization. We furthermore study the link between effectiveness and efficiency, on in-domain and zero-shot settings, leading to state-of-the-art results in both scenarios for sufficiently expressive models.
arXiv commentAccepted at SIGIR22 as a short paper (this work is the extension of SPLADE v2)
Cited by in corpus (5)
- Adaptive Re-Ranking with a Corpus Graph
- Foundations of Vector Retrieval
- Optimizing Guided Traversal for Fast Learned Sparse Retrieval
- Representation Sparsification with Hybrid Thresholding for Fast SPLADE-based Document Retrieval
- PEFA: Parameter-Free Adapters for Large-scale Embedding-based Retrieval Models