Path-Normalized Optimization of Recurrent Neural Networks with ReLU Activations
arXiv:1605.07154
Abstract
We investigate the parameter-space geometry of recurrent neural networks (RNNs), and develop an adaptation of path-SGD optimization method, attuned to this geometry, that can learn plain RNNs with ReLU activations. On several datasets that require capturing long-term dependency structure, we show that path-SGD can significantly improve trainability of ReLU RNNs compared to RNNs trained with SGD, even with various recently suggested initialization schemes.
arXiv comment15 pages
References in corpus (5)
- Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
- A Simple Way to Initialize Recurrent Networks of Rectified Linear Units
- Unitary Evolution Recurrent Neural Networks
- Architectural Complexity Measures of Recurrent Neural Networks
- Regularization and nonlinearities for neural language models: when are they needed?