Strongly-Typed Recurrent Neural Networks

Abstract

Recurrent neural networks are increasing popular models for sequential learning. Unfortunately, although the most effective RNN architectures are perhaps excessively complicated, extensive searches have not found simpler alternatives. This paper imports ideas from physics and functional programming into RNN design to provide guiding principles. From physics, we introduce type constraints, analogous to the constraints that forbids adding meters to seconds. From functional programming, we require that strongly-typed architectures factorize into stateless learnware and state-dependent firmware, reducing the impact of side-effects. The features learned by strongly-typed nets have a simple semantic interpretation via dynamic average-pooling on one-dimensional convolutions. We also show that strongly-typed gradients are better behaved than in classical architectures, and characterize the representational power of strongly-typed nets. Finally, experiments show that, despite being more constrained, strongly-typed architectures achieve lower training and comparable generalization error to classical architectures.

10 pages, final version, ICML 2016

References in corpus (7)

• Sequence to Sequence Learning with Neural Networks
• LSTM: A Search Space Odyssey
• On the difficulty of training Recurrent Neural Networks
• Recurrent Neural Network Regularization
• Visualizing and Understanding Recurrent Networks
• A Simple Way to Initialize Recurrent Networks of Rectified Linear Units
• Learning to Transduce with Unbounded Memory

Cited by in corpus (8)

• Neural Abstractive Text Summarization with Sequence-to-Sequence Models
• Sliced Recurrent Neural Networks
• Parallelizing Legendre Memory Unit Training
• Deep Online Convex Optimization with Gated Games
• Benchmarking Deep Sequential Models on Volatility Predictions for Financial Time Series
• The Nonlinearity Coefficient - A Practical Guide to Neural Architecture Design
• Multi-Scale Quasi-RNN for Next Item Recommendation
• Rational Recurrences