Modularized Transfomer-based Ranking Framework
arXiv:2004.13313
Abstract
Recent innovations in Transformer-based ranking models have advanced the state-of-the-art in information retrieval. However, these Transformers are computationally expensive, and their opaque hidden states make it hard to understand the ranking process. In this work, we modularize the Transformer ranker into separate modules for text representation and interaction. We show how this design enables substantially faster ranking using offline pre-computed representations and light-weight online interactions. The modular design is also easier to interpret and sheds light on the ranking process in Transformer rankers.
References in corpus (13)
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- Distilling the Knowledge in a Neural Network
- Convolutional Neural Network Architectures for Matching Natural Language Sentences
- A Deep Relevance Matching Model for Ad-hoc Retrieval
- End-to-End Neural Ad-hoc Ranking with Kernel Pooling
- Generating Long Sequences with Sparse Transformers
- Deeper Text Understanding for IR with Contextual Neural Language Modeling
- Passage Re-ranking with BERT
- Document Expansion by Query Prediction
- Understanding the Behaviors of BERT in Ranking
- Context-Aware Sentence/Passage Term Importance Estimation For First Stage Retrieval
- Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned
- Overview of the TREC 2021 deep learning track