Compressive Transformers for Long-Range Sequence Modelling
arXiv:1911.05507
Abstract
We present the Compressive Transformer, an attentive sequence model which compresses past memories for long-range sequence learning. We find the Compressive Transformer obtains state-of-the-art language modelling results in the WikiText-103 and Enwik8 benchmarks, achieving 17.1 ppl and 0.97 bpc respectively. We also find it can model high-frequency speech effectively and can be used as a memory mechanism for RL, demonstrated on an object matching task. To promote the domain of long-range sequence learning, we propose a new open-vocabulary language modelling benchmark derived from books, PG-19.
19 pages, 6 figures, 10 tables
References in corpus (16)
- Adam: A Method for Stochastic Optimization
- The Curious Case of Neural Text Degeneration
- Language Modeling with Gated Convolutional Networks
- Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
- The Goldilocks Principle: Reading Children's Books with Explicit Memory Representations
- Generating Long Sequences with Sparse Transformers
- Pointer Sentinel Mixture Models
- Quasi-Recurrent Neural Networks
- Neural Machine Translation in Linear Time
- Hierarchical Multiscale Recurrent Neural Networks
- Trellis Networks for Sequence Modeling
- Multiplicative LSTM for sequence modelling
- Scaling Memory-Augmented Neural Networks with Sparse Reads and Writes
- Adaptive Input Representations for Neural Language Modeling
- Dynamic Evaluation of Transformer Language Models
- Fast Parametric Learning with Activation Memorization
Cited by in corpus (18)
- Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting
- Big Bird: Transformers for Longer Sequences
- Focal Self-attention for Local-Global Interactions in Vision Transformers
- Training with Quantization Noise for Extreme Model Compression
- CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windows
- Beyond 512 Tokens: Siamese Multi-depth Transformer-based Hierarchical Encoder for Long-Form Document Matching
- Chasing Sparsity in Vision Transformers: An End-to-End Exploration
- Addressing Some Limitations of Transformers with Feedback Memory
- ETC: Encoding Long and Structured Inputs in Transformers
- Cluster-Former: Clustering-based Sparse Transformer for Long-Range Dependency Encoding
- Exploring Transformers for Large-Scale Speech Recognition
- Dynamic Memory based Attention Network for Sequential Recommendation
- What Context Features Can Transformer Language Models Use?
- ReadTwice: Reading Very Large Documents with Memories
- Current Limitations of Language Models: What You Need is Retrieval
- Do Transformers Need Deep Long-Range Memory
- Large Product Key Memory for Pretrained Language Models
- Benchmarking down-scaled (not so large) pre-trained language models