Representation Sparsification with Hybrid Thresholding for Fast SPLADE-based Document Retrieval

Abstract

Learned sparse document representations using a transformer-based neural model has been found to be attractive in both relevance effectiveness and time efficiency. This paper describes a representation sparsification scheme based on hard and soft thresholding with an inverted index approximation for faster SPLADE-based document retrieval. It provides analytical and experimental results on the impact of this learnable hybrid thresholding scheme.

arXiv commentThis paper is published in SIGIR'23

References in corpus (13)

• Decoding billions of integers per second through vectorization
• Overview of the TREC 2020 deep learning track
• Soft Threshold Weight Reparameterization for Learnable Sparsity
• An Efficiency Study for SPLADE Models
• A Few Brief Notes on DeepImpact, COIL, and a Conceptual Framework for Information Retrieval Techniques
• Improving Efficient Neural Ranking Models with Cross-Architecture Knowledge Distillation
• Optimizing Guided Traversal for Fast Learned Sparse Retrieval
• From Distillation to Hard Negative Sampling: Making Sparse Neural IR Models More Effective
• Learning Discrete Representations via Constrained Clustering for Effective and Efficient Dense Retrieval
• Sparsifying Sparse Representations for Passage Retrieval by Top-$k$ Masking
• Distill-VQ: Learning Retrieval Oriented Vector Quantization By Distilling Knowledge from Dense Embeddings
• SPDY: Accurate Pruning with Speedup Guarantees
• Dual Skipping Guidance for Document Retrieval with Learned Sparse Representations

Cited by in corpus (1)

• Adaptive Sparsity Optimization with Learnable Soft Top-K and Per-Term Thresholding for Efficient Retrieval