The Power of Selecting Key Blocks with Local Pre-ranking for Long Document Information Retrieval
arXiv:2111.09852 · doi:10.1145/3568394
Abstract
On a wide range of natural language processing and information retrieval tasks, transformer-based models, particularly pre-trained language models like BERT, have demonstrated tremendous effectiveness. Due to the quadratic complexity of the self-attention mechanism, however, such models have difficulties processing long documents. Recent works dealing with this issue include truncating long documents, in which case one loses potential relevant information, segmenting them into several passages, which may lead to miss some information and high computational complexity when the number of passages is large, or modifying the self-attention mechanism to make it sparser as in sparse-attention models, at the risk again of missing some information. We follow here a slightly different approach in which one first selects key blocks of a long document by local query-block pre-ranking, and then few blocks are aggregated to form a short document that can be processed by a model such as BERT. Experiments conducted on standard Information Retrieval datasets demonstrate the effectiveness of the proposed approach.
arXiv comment34 pages, accepted by ACM Transactions on Information Systems (TOIS)
References in corpus (26)
- Scikit-learn: Machine Learning in Python
- Distributed Representations of Words and Phrases and their Compositionality
- Representation Learning with Contrastive Predictive Coding
- HuggingFace's Transformers: State-of-the-art Natural Language Processing
- Longformer: The Long-Document Transformer
- Mixed Precision Training
- Convolutional Neural Network Architectures for Matching Natural Language Sentences
- A Deep Relevance Matching Model for Ad-hoc Retrieval
- Deep Sentence Embedding Using Long Short-Term Memory Networks: Analysis and Application to Information Retrieval
- End-to-End Neural Ad-hoc Ranking with Kernel Pooling
- Generating Long Sequences with Sparse Transformers
- Deeper Text Understanding for IR with Contextual Neural Language Modeling
- Passage Re-ranking with BERT
- Big Bird: Transformers for Longer Sequences
- CEDR: Contextualized Embeddings for Document Ranking
- DeepRank: A New Deep Architecture for Relevance Ranking in Information Retrieval
- Introducing LETOR 4.0 Datasets
- Match-SRNN: Modeling the Recursive Matching Structure with Spatial RNN
- Overview of the TREC 2020 deep learning track
- Modeling Diverse Relevance Patterns in Ad-hoc Retrieval
- CondenseNet: An Efficient DenseNet using Learned Group Convolutions
- Balanced Sparsity for Efficient DNN Inference on GPU
- Overview of the TREC 2019 deep learning track
- PARADE: Passage Representation Aggregation for Document Reranking
- Neural Passage Retrieval with Improved Negative Contrast
- Intra-Document Cascading: Learning to Select Passages for Neural Document Ranking