Kronecker Recurrent Units

Abstract

Our work addresses two important issues with recurrent neural networks: (1) they are over-parameterized, and (2) the recurrence matrix is ill-conditioned. The former increases the sample complexity of learning and the training time. The latter causes the vanishing and exploding gradient problem. We present a flexible recurrent neural network model called Kronecker Recurrent Units (KRU). KRU achieves parameter efficiency in RNNs through a Kronecker factored recurrent matrix. It overcomes the ill-conditioning of the recurrent matrix by enforcing soft unitary constraints on the factors. Thanks to the small dimensionality of the factors, maintaining these constraints is computationally efficient. Our experimental results on seven standard data-sets reveal that KRU can reduce the number of parameters by three orders of magnitude in the recurrent weight matrix compared to the existing recurrent models, without trading the statistical performance. These results in particular show that while there are advantages in having a high dimensional recurrent space, the capacity of the recurrent part of the model can be dramatically reduced.

References in corpus (13)

• Adam: A Method for Stochastic Optimization
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
• Do Deep Nets Really Need to be Deep?
• Compressing Neural Networks with the Hashing Trick
• A Simple Way to Initialize Recurrent Networks of Rectified Linear Units
• Modeling Temporal Dependencies in High-Dimensional Sequences: Application to Polyphonic Music Generation and Transcription
• Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximation
• Optimizing Neural Networks with Kronecker-factored Approximate Curvature
• On orthogonality and learning recurrent networks with long term dependencies
• Gradient Descent Learns Linear Dynamical Systems
• Efficient Orthogonal Parametrisation of Recurrent Neural Networks Using Householder Reflections
• Low-rank passthrough neural networks

Cited by in corpus (5)

• Compressing RNNs for IoT devices by 15-38x using Kronecker Products
• Dynamical Isometry and a Mean Field Theory of LSTMs and GRUs
• One Size Does Not Fit All: Multi-Scale, Cascaded RNNs for Radar Classification
• Implementation of Optical Deep Neural Networks using the Fabry-Perot Interferometer
• RNN Training along Locally Optimal Trajectories via Frank-Wolfe Algorithm