On the State of the Art of Evaluation in Neural Language Models
arXiv:1707.05589
Abstract
Ongoing innovations in recurrent neural network architectures have provided a steady influx of apparently state-of-the-art results on language modelling benchmarks. However, these have been evaluated using differing code bases and limited computational resources, which represent uncontrolled sources of experimental variation. We reevaluate several popular architectures and regularisation methods with large-scale automatic black-box hyperparameter tuning and arrive at the somewhat surprising conclusion that standard LSTM architectures, when properly regularised, outperform more recent models. We establish a new state of the art on the Penn Treebank and Wikitext-2 corpora, as well as strong baselines on the Hutter Prize dataset.
References in corpus (6)
- Adam: A Method for Stochastic Optimization
- Neural Architecture Search with Reinforcement Learning
- Long Short-Term Memory Based Recurrent Neural Network Architectures for Large Vocabulary Speech Recognition
- Pointer Sentinel Mixture Models
- Grid Long Short-Term Memory
- Dynamic Evaluation of Neural Sequence Models
Cited by in corpus (15)
- Deep learning for molecular design - a review of the state of the art
- Beyond Word Importance: Contextual Decomposition to Extract Interactions from LSTMs
- Gmail Smart Compose: Real-Time Assisted Writing
- Neural Text Generation: A Practical Guide
- The Implicit and Explicit Regularization Effects of Dropout
- Strong Baselines for Simple Question Answering over Knowledge Graphs with and without Neural Networks
- Differentiable Architecture Search with Ensemble Gumbel-Softmax
- Can LSTM Learn to Capture Agreement? The Case of Basque
- Efficient Contextualized Representation: Language Model Pruning for Sequence Labeling
- Improving Neural Machine Translation with Pre-trained Representation
- Variational Smoothing in Recurrent Neural Network Language Models
- WeNet: Weighted Networks for Recurrent Network Architecture Search
- The Effectiveness of Multitask Learning for Phenotyping with Electronic Health Records Data
- Measuring the Complexity of Domains Used to Evaluate AI Systems
- Mutual Information Constraints for Monte-Carlo Objectives