Dynamic Evaluation of Neural Sequence Models

Abstract

We present methodology for using dynamic evaluation to improve neural sequence models. Models are adapted to recent history via a gradient descent based mechanism, causing them to assign higher probabilities to re-occurring sequential patterns. Dynamic evaluation outperforms existing adaptation approaches in our comparisons. Dynamic evaluation improves the state-of-the-art word-level perplexities on the Penn Treebank and WikiText-2 datasets to 51.1 and 44.3 respectively, and the state-of-the-art character-level cross-entropies on the text8 and Hutter Prize datasets to 1.19 bits/char and 1.08 bits/char respectively.

References in corpus (8)

• Neural Architecture Search with Reinforcement Learning
• Recurrent Neural Network Regularization
• Pointer Sentinel Mixture Models
• Regularizing and Optimizing LSTM Language Models
• Neural Machine Translation in Linear Time
• Hierarchical Multiscale Recurrent Neural Networks
• Improving Neural Language Models with a Continuous Cache
• Fast-Slow Recurrent Neural Networks

Cited by in corpus (4)

• On the State of the Art of Evaluation in Neural Language Models
• Deep Online Learning via Meta-Learning: Continual Adaptation for Model-Based RL
• WeNet: Weighted Networks for Recurrent Network Architecture Search
• Character n-gram Embeddings to Improve RNN Language Models