Dynamic Evaluation of Transformer Language Models
arXiv:1904.08378
Abstract
This research note combines two methods that have recently improved the state of the art in language modeling: Transformers and dynamic evaluation. Transformers use stacked layers of self-attention that allow them to capture long range dependencies in sequential data. Dynamic evaluation fits models to the recent sequence history, allowing them to assign higher probabilities to re-occurring sequential patterns. By applying dynamic evaluation to Transformer-XL models, we improve the state of the art on enwik8 from 0.99 to 0.94 bits/char, text8 from 1.08 to 1.04 bits/char, and WikiText-103 from 18.3 to 16.4 perplexity points.
References in corpus (4)
Cited by in corpus (15)
- Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
- VLT: Vision-Language Transformer and Query Generation for Referring Segmentation
- Generalization through Memorization: Nearest Neighbor Language Models
- Compressive Transformers for Long-Range Sequence Modelling
- Addressing Some Limitations of Transformers with Feedback Memory
- Mind the Gap: Assessing Temporal Generalization in Neural Language Models
- GNN-LM: Language Modeling based on Global Contexts via GNN
- Vision-Language Transformer and Query Generation for Referring Segmentation
- Neural Machine Translation: A Review and Survey
- Behavior Gated Language Models
- Adaptive Semiparametric Language Models
- A Simple Recurrent Unit with Reduced Tensor Product Representations
- Temporal Adaptation of BERT and Performance on Downstream Document Classification: Insights from Social Media
- Context-aware RNNLM Rescoring for Conversational Speech Recognition
- SimpleBooks: Long-term dependency book dataset with simplified English vocabulary for word-level language modeling