Low-rank passthrough neural networks
arXiv:1603.03116
Abstract
Various common deep learning architectures, such as LSTMs, GRUs, Resnets and Highway Networks, employ state passthrough connections that support training with high feed-forward depth or recurrence over many time steps. These "Passthrough Networks" architectures also enable the decoupling of the network state size from the number of parameters of the network, a possibility has been studied by \newcite{Sak2014} with their low-rank parametrization of the LSTM. In this work we extend this line of research, proposing effective, low-rank and low-rank plus diagonal matrix parametrizations for Passthrough Networks which exploit this decoupling property, reducing the data complexity and memory requirements of the network while preserving its memory capacity. This is particularly beneficial in low-resource settings as it supports expressive models with a compact parametrization less susceptible to overfitting. We present competitive experimental results on several tasks, including language modeling and a near state of the art result on sequential randomly-permuted MNIST classification, a hard task on natural data.
12 pages, 2 figures
References in corpus (11)
- Recurrent Neural Network Regularization
- DRAW: A Recurrent Neural Network For Image Generation
- Exploring the Limits of Language Modeling
- A Theoretically Grounded Application of Dropout in Recurrent Neural Networks
- A Simple Way to Initialize Recurrent Networks of Rectified Linear Units
- Grammar as a Foreign Language
- Neural GPUs Learn Algorithms
- Unitary Evolution Recurrent Neural Networks
- Character-based Neural Machine Translation
- Architectural Complexity Measures of Recurrent Neural Networks
- Neural Programmer: Inducing Latent Programs with Gradient Descent