Investigating the Effects of Sparse Attention on Cross-Encoders

Abstract

Cross-encoders are effective passage and document re-rankers but less efficient than other neural or classic retrieval models. A few previous studies have applied windowed self-attention to make cross-encoders more efficient. However, these studies did not investigate the potential and limits of different attention patterns or window sizes. We close this gap and systematically analyze how token interactions can be reduced without harming the re-ranking effectiveness. Experimenting with asymmetric attention and different window sizes, we find that the query tokens do not need to attend to the passage or document tokens for effective re-ranking and that very small window sizes suffice. In our experiments, even windows of 4 tokens still yield effectiveness on par with previous cross-encoders while reducing the memory requirements by at least 22% / 59% and being 1% / 43% faster at inference time for passages / documents.

Accepted at ECIR'24

References in corpus (10)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Deeper Text Understanding for IR with Contextual Neural Language Modeling
• Multi-Stage Document Ranking with BERT
• FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
• Simplified Data Wrangling with ir_datasets
• Efficient Document Re-Ranking for Transformers by Precomputing Term Representations
• The Information Retrieval Experiment Platform
• Extending Context Window of Large Language Models via Positional Interpolation
• HLATR: Enhance Multi-stage Text Retrieval with Hybrid List Aware Transformer Reranking
• In Defense of Cross-Encoders for Zero-Shot Retrieval