Long Document Ranking with Query-Directed Sparse Transformer
arXiv:2010.12683
Abstract
The computing cost of transformer self-attention often necessitates breaking long documents to fit in pretrained models in document ranking tasks. In this paper, we design Query-Directed Sparse attention that induces IR-axiomatic structures in transformer self-attention. Our model, QDS-Transformer, enforces the principle properties desired in ranking: local contextualization, hierarchical representation, and query-oriented proximity matching, while it also enjoys efficiency from sparsity. Experiments on one fully supervised and three few-shot TREC document ranking benchmarks demonstrate the consistent and robust advantage of QDS-Transformer over previous approaches, as they either retrofit long documents into BERT or use sparse attention without emphasizing IR principles. We further quantify the computing complexity and demonstrates that our sparse attention with TVM implementation is twice more efficient than the fully-connected self-attention. All source codes, trained model, and predictions of this work are available at https://github.com/hallogameboy/QDS-Transformer.
arXiv commentAccepted by EMNLP 2020, 12 pages, 5 figures
References in corpus (14)
- Adam: A Method for Stochastic Optimization
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- Longformer: The Long-Document Transformer
- A Deep Relevance Matching Model for Ad-hoc Retrieval
- End-to-End Neural Ad-hoc Ranking with Kernel Pooling
- Generating Long Sequences with Sparse Transformers
- Deeper Text Understanding for IR with Contextual Neural Language Modeling
- Passage Re-ranking with BERT
- Multi-Stage Document Ranking with BERT
- TVM: An Automated End-to-End Optimizing Compiler for Deep Learning
- Simple Applications of BERT for Ad Hoc Document Retrieval
- Overview of the TREC 2022 deep learning track
- A Deep Look into Neural Ranking Models for Information Retrieval
- TU Wien @ TREC Deep Learning '19 -- Simple Contextualization for Re-ranking