ReadTwice: Reading Very Large Documents with Memories
arXiv:2105.04241
Abstract
Knowledge-intensive tasks such as question answering often require assimilating information from different sections of large inputs such as books or article collections. We propose ReadTwice, a simple and effective technique that combines several strengths of prior approaches to model long-range dependencies with Transformers. The main idea is to read text in small segments, in parallel, summarizing each segment into a memory table to be used in a second read of the text. We show that the method outperforms models of comparable size on several question answering (QA) datasets and sets a new state of the art on the challenging NarrativeQA task, with questions about entire books. Source code and pre-trained checkpoints for ReadTwice can be found at https://goo.gle/research-readtwice.
arXiv commentTo appear in the proceedings of NAACL 2021
References in corpus (8)
- Longformer: The Long-Document Transformer
- ERNIE: Enhanced Representation through Knowledge Integration
- Leveraging Pre-trained Checkpoints for Sequence Generation Tasks
- Big Bird: Transformers for Longer Sequences
- Relevance of Unsupervised Metrics in Task-Oriented Dialogue for Evaluating Natural Language Generation
- Compressive Transformers for Long-Range Sequence Modelling
- Language Model Pre-training for Hierarchical Document Representations
- Probabilistic Assumptions Matter: Improved Models for Distantly-Supervised Document-Level Question Answering