ETC: Encoding Long and Structured Inputs in Transformers
arXiv:2004.08483
Abstract
Transformer models have advanced the state of the art in many Natural Language Processing (NLP) tasks. In this paper, we present a new Transformer architecture, Extended Transformer Construction (ETC), that addresses two key challenges of standard Transformer architectures, namely scaling input length and encoding structured inputs. To scale attention to longer inputs, we introduce a novel global-local attention mechanism between global tokens and regular input tokens. We also show that combining global-local attention with relative position encodings and a Contrastive Predictive Coding (CPC) pre-training objective allows ETC to encode structured inputs. We achieve state-of-the-art results on four natural language datasets requiring long and/or structured inputs.
arXiv commentAccepted at EMNLP 2020
References in corpus (22)
- Representation Learning with Contrastive Predictive Coding
- Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
- Longformer: The Long-Document Transformer
- SQuAD: 100,000+ Questions for Machine Comprehension of Text
- Generating Long Sequences with Sparse Transformers
- Leveraging Pre-trained Checkpoints for Sequence Generation Tasks
- Big Bird: Transformers for Longer Sequences
- Practical and Optimal LSH for Angular Distance
- The Reversible Residual Network: Backpropagation Without Storing Activations
- Adafactor: Adaptive Learning Rates with Sublinear Memory Cost
- HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering
- RACE: Large-scale ReAding Comprehension Dataset From Examinations
- A BERT Baseline for the Natural Questions
- Bi-Directional Block Self-Attention for Fast and Memory-Efficient Sequence Modeling
- Generating Wikipedia by Summarizing Long Sequences
- BP-Transformer: Modelling Long-Range Context via Binary Partitioning
- HIBERT: Document Level Pre-training of Hierarchical Bidirectional Transformers for Document Summarization
- Compressive Transformers for Long-Range Sequence Modelling
- Adaptively Sparse Transformers
- RikiNet: Reading Wikipedia Pages for Natural Question Answering
- Generating Logical Forms from Graph Representations of Text and Entities
- Open Domain Web Keyphrase Extraction Beyond Language Modeling
Cited by in corpus (10)
- Long Range Arena: A Benchmark for Efficient Transformers
- CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representation
- Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers
- Generative Adversarial Transformers
- Pre-Trained Models: Past, Present and Future
- Length-Adaptive Transformer: Train Once with Length Drop, Use Anytime with Search
- On Learning the Transformer Kernel
- Global memory transformer for processing long documents
- Interpretable Self-supervised Multi-task Learning for COVID-19 Information Retrieval and Extraction
- Iterative Decoding for Compositional Generalization in Transformers