Set-Encoder: Permutation-Invariant Inter-Passage Attention for Listwise Passage Re-Ranking with Cross-Encoders

Abstract

Existing cross-encoder models can be categorized as pointwise, pairwise, or listwise. Pairwise and listwise models allow passage interactions, which typically makes them more effective than pointwise models but less efficient and less robust to input passage order permutations. To enable efficient permutation-invariant passage interactions during re-ranking, we propose a new cross-encoder architecture with inter-passage attention: the Set-Encoder. In experiments on TREC Deep Learning and TIREx, the Set-Encoder is as effective as state-of-the-art listwise models while being more efficient and invariant to input passage order permutations. Compared to pointwise models, the Set-Encoder is particularly more effective when considering inter-passage information, such as novelty, and retains its advantageous properties compared to other listwise models. Our code is publicly available at https://github.com/webis-de/ECIR-25.

Accepted at ECIR'25

References in corpus (18)

• SciPy 1.0--Fundamental Algorithms for Scientific Computing in Python
• Array Programming with NumPy
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Representation Learning with Contrastive Predictive Coding
• HuggingFace's Transformers: State-of-the-art Natural Language Processing
• CORD-19: The COVID-19 Open Research Dataset
• Multi-Stage Document Ranking with BERT
• Simplified Data Wrangling with ir_datasets
• The Information Retrieval Experiment Platform
• RankFormer: Listwise Learning-to-Rank Using Listwide Labels
• RankZephyr: Effective and Robust Zero-Shot Listwise Reranking is a Breeze!
• RankVicuna: Zero-Shot Listwise Document Reranking with Open-Source Large Language Models
• Rank-DistiLLM: Closing the Effectiveness Gap Between Cross-Encoders and LLMs for Passage Re-Ranking
• Sparse Pairwise Re-ranking with Pre-trained Transformers
• Found in the Middle: Permutation Self-Consistency Improves Listwise Ranking in Large Language Models
• Lightning IR: Straightforward Fine-tuning and Inference of Transformer-based Language Models for Information Retrieval
• RankT5: Fine-Tuning T5 for Text Ranking with Ranking Losses
• Scaling Down, LiTting Up: Efficient Zero-Shot Listwise Reranking with Seq2seq Encoder-Decoder Models