Distilling Dense Representations for Ranking using Tightly-Coupled Teachers

Abstract

We present an approach to ranking with dense representations that applies knowledge distillation to improve the recently proposed late-interaction ColBERT model. Specifically, we distill the knowledge from ColBERT's expressive MaxSim operator for computing relevance scores into a simple dot product, thus enabling single-step ANN search. Our key insight is that during distillation, tight coupling between the teacher model and the student model enables more flexible distillation strategies and yields better learned representations. We empirically show that our approach improves query latency and greatly reduces the onerous storage requirements of ColBERT, while only making modest sacrifices in terms of effectiveness. By combining our dense representations with sparse representations derived from document expansion, we are able to approach the effectiveness of a standard cross-encoder reranker using BERT that is orders of magnitude slower.

References in corpus (9)

• Distilling the Knowledge in a Neural Network
• REALM: Retrieval-Augmented Language Model Pre-Training
• Passage Re-ranking with BERT
• Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval
• Pretrained Transformers for Text Ranking: BERT and Beyond
• Pre-training Tasks for Embedding-based Large-scale Retrieval
• Sparse, Dense, and Attentional Representations for Text Retrieval
• Complementing Lexical Retrieval with Semantic Residual Embedding
• Overview of the TREC 2021 deep learning track

Cited by in corpus (24)

• Information Retrieval: Recent Advances and Beyond
• Semantic Models for the First-stage Retrieval: A Comprehensive Review
• Pretrained Transformers for Text Ranking: BERT and Beyond
• Rethinking Search: Making Domain Experts out of Dilettantes
• Generative Relevance Feedback with Large Language Models
• The Expando-Mono-Duo Design Pattern for Text Ranking with Pretrained Sequence-to-Sequence Models
• Pyserini: An Easy-to-Use Python Toolkit to Support Replicable IR Research with Sparse and Dense Representations
• A Replication Study of Dense Passage Retriever
• CharacterBERT and Self-Teaching for Improving the Robustness of Dense Retrievers on Queries with Typos
• To Interpolate or not to Interpolate: PRF, Dense and Sparse Retrievers
• SpaDE: Improving Sparse Representations using a Dual Document Encoder for First-stage Retrieval
• Optimizing Dense Retrieval Model Training with Hard Negatives
• CODER: An efficient framework for improving retrieval through COntextual Document Embedding Reranking
• Implicit Feedback for Dense Passage Retrieval: A Counterfactual Approach
• Condenser: a Pre-training Architecture for Dense Retrieval
• ZeQR: Zero-shot Query Reformulation for Conversational Search
• Contextual Fine-to-Coarse Distillation for Coarse-grained Response Selection in Open-Domain Conversations
• Hybrid and Collaborative Passage Reranking
• SPLADE: Sparse Lexical and Expansion Model for First Stage Ranking
• Contextualized Query Embeddings for Conversational Search
• Improving Query Representations for Dense Retrieval with Pseudo Relevance Feedback
• Jointly Optimizing Query Encoder and Product Quantization to Improve Retrieval Performance
• Quality and Cost Trade-offs in Passage Re-ranking Task
• Annotative Indexing