Delaying Interaction Layers in Transformer-based Encoders for Efficient Open Domain Question Answering

Abstract

Open Domain Question Answering (ODQA) on a large-scale corpus of documents (e.g. Wikipedia) is a key challenge in computer science. Although transformer-based language models such as Bert have shown on SQuAD the ability to surpass humans for extracting answers in small passages of text, they suffer from their high complexity when faced to a much larger search space. The most common way to tackle this problem is to add a preliminary Information Retrieval step to heavily filter the corpus and only keep the relevant passages. In this paper, we propose a more direct and complementary solution which consists in applying a generic change in the architecture of transformer-based models to delay the attention between subparts of the input and allow a more efficient management of computations. The resulting variants are competitive with the original models on the extractive task and allow, on the ODQA setting, a significant speedup and even a performance improvement in many cases.

References in corpus (7)

• Distributed Representations of Words and Phrases and their Compositionality
• A Deep Relevance Matching Model for Ad-hoc Retrieval
• End-to-End Open-Domain Question Answering with BERTserini
• Passage Re-ranking with BERT
• Big Bird: Transformers for Longer Sequences
• Simple Applications of BERT for Ad Hoc Document Retrieval
• R${}^3$: Reinforced Reader-Ranker for Open-Domain Question Answering

Cited by in corpus (1)

• Learning Dense Representations of Phrases at Scale