Dynamic Evaluation of Neural Sequence Models
arXiv:1709.07432
Abstract
We present methodology for using dynamic evaluation to improve neural sequence models. Models are adapted to recent history via a gradient descent based mechanism, causing them to assign higher probabilities to re-occurring sequential patterns. Dynamic evaluation outperforms existing adaptation approaches in our comparisons. Dynamic evaluation improves the state-of-the-art word-level perplexities on the Penn Treebank and WikiText-2 datasets to 51.1 and 44.3 respectively, and the state-of-the-art character-level cross-entropies on the text8 and Hutter Prize datasets to 1.19 bits/char and 1.08 bits/char respectively.
References in corpus (14)
- Adam: A Method for Stochastic Optimization
- Neural Architecture Search with Reinforcement Learning
- Recurrent Neural Network Regularization
- Character-Aware Neural Language Models
- Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks
- Pointer Sentinel Mixture Models
- Regularizing and Optimizing LSTM Language Models
- Neural Machine Translation in Linear Time
- Hierarchical Multiscale Recurrent Neural Networks
- Bayesian Recurrent Neural Networks
- Improving Neural Language Models with a Continuous Cache
- Multiplicative LSTM for sequence modelling
- On Multiplicative Integration with Recurrent Neural Networks
- Fast-Slow Recurrent Neural Networks
Cited by in corpus (8)
- Adaptation Algorithms for Neural Network-Based Speech Recognition: An Overview
- Lifelong Incremental Reinforcement Learning with Online Bayesian Inference
- On the State of the Art of Evaluation in Neural Language Models
- Deep Online Learning via Meta-Learning: Continual Adaptation for Model-Based RL
- A Dirichlet Process Mixture of Robust Task Models for Scalable Lifelong Reinforcement Learning
- Contextual Temperature for Language Modeling
- WeNet: Weighted Networks for Recurrent Network Architecture Search
- Character n-gram Embeddings to Improve RNN Language Models