Rank-DistiLLM: Closing the Effectiveness Gap Between Cross-Encoders and LLMs for Passage Re-Ranking
arXiv:2405.07920 · doi:10.1007/978-3-031-88714-7_31
Abstract
Cross-encoders distilled from large language models (LLMs) are often more effective re-rankers than cross-encoders fine-tuned on manually labeled data. However, distilled models do not match the effectiveness of their teacher LLMs. We hypothesize that this effectiveness gap is due to the fact that previous work has not applied the best-suited methods for fine-tuning cross-encoders on manually labeled data (e.g., hard-negative sampling, deep sampling, and listwise loss functions). To close this gap, we create a new dataset, Rank-DistiLLM. Cross-encoders trained on Rank-DistiLLM achieve the effectiveness of LLMs while being up to 173 times faster and 24 times more memory efficient. Our code and data is available at https://github.com/webis-de/ECIR-25.
arXiv commentAccepted at ECIR'25
References in corpus (17)
- SciPy 1.0--Fundamental Algorithms for Scientific Computing in Python
- Array Programming with NumPy
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- Representation Learning with Contrastive Predictive Coding
- HuggingFace's Transformers: State-of-the-art Natural Language Processing
- CORD-19: The COVID-19 Open Research Dataset
- Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval
- FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
- Simplified Data Wrangling with ir_datasets
- Improving Efficient Neural Ranking Models with Cross-Architecture Knowledge Distillation
- The Information Retrieval Experiment Platform
- RankZephyr: Effective and Robust Zero-Shot Listwise Reranking is a Breeze!
- RankVicuna: Zero-Shot Listwise Document Reranking with Open-Source Large Language Models
- Lightning IR: Straightforward Fine-tuning and Inference of Transformer-based Language Models for Information Retrieval
- In Defense of Cross-Encoders for Zero-Shot Retrieval
- RankT5: Fine-Tuning T5 for Text Ranking with Ranking Losses
- Scaling Down, LiTting Up: Efficient Zero-Shot Listwise Reranking with Seq2seq Encoder-Decoder Models