Learning Simpler Language Models with the Differential State Framework
arXiv:1703.08864
Abstract
Learning useful information across long time lags is a critical and difficult problem for temporal neural models in tasks such as language modeling. Existing architectures that address the issue are often complex and costly to train. The Differential State Framework (DSF) is a simple and high-performing design that unifies previously introduced gated neural models. DSF models maintain longer-term memory by learning to interpolate between a fast-changing data-driven representation and a slowly changing, implicitly stable state. This requires hardly any more parameters than a classical, simple recurrent network. Within the DSF framework, a new architecture is presented, the Delta-RNN. In language modeling at the word and character levels, the Delta-RNN outperforms popular complex architectures, such as the Long Short Term Memory (LSTM) and the Gated Recurrent Unit (GRU), and, when regularized, performs comparably to several state-of-the-art baselines. At the subword level, the Delta-RNN's performance is comparable to that of complex gated architectures.
arXiv commentEdits/revisions applied throughout document
References in corpus (14)
- Adam: A Method for Stochastic Optimization
- Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
- Recurrent Neural Network Regularization
- A Simple Way to Initialize Recurrent Networks of Rectified Linear Units
- A Clockwork RNN
- Hierarchical Multiscale Recurrent Neural Networks
- Learning Longer Memory in Recurrent Neural Networks
- Zoneout: Regularizing RNNs by Randomly Preserving Hidden Activations
- Noisy Activation Functions
- On Multiplicative Integration with Recurrent Neural Networks
- Recurrent Batch Normalization
- Larger-Context Language Modelling
- Memory Augmented Neural Networks with Wormhole Connections
- Variable Computation in Recurrent Neural Networks