MinimalRNN: Toward More Interpretable and Trainable Recurrent Neural Networks
arXiv:1711.06788
Abstract
We introduce MinimalRNN, a new recurrent neural network architecture that achieves comparable performance as the popular gated RNNs with a simplified structure. It employs minimal updates within RNN, which not only leads to efficient learning and testing but more importantly better interpretability and trainability. We demonstrate that by endorsing the more restrictive update rule, MinimalRNN learns disentangled RNN states. We further examine the learning dynamics of different RNN structures using input-output Jacobians, and show that MinimalRNN is able to capture longer range dependencies than existing RNN architectures.
arXiv commentPresented at NIPS 2017 Symposium on Interpretable Machine Learning
References in corpus (7)
- Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
- Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
- Exploring the Limits of Language Modeling
- Visualizing and Understanding Recurrent Networks
- Session-based Recommendations with Recurrent Neural Networks
- All You Need is Beyond a Good Init: Exploring Better Solution for Training Extremely Deep Convolutional Neural Networks with Orthonormality and Modulation
- A recurrent neural network without chaos