A Simple Way to Initialize Recurrent Networks of Rectified Linear Units

Abstract

Learning long term dependencies in recurrent networks is difficult due to vanishing and exploding gradients. To overcome this difficulty, researchers have developed sophisticated optimization techniques and network architectures. In this paper, we propose a simpler solution that use recurrent neural networks composed of rectified linear units. Key to our solution is the use of the identity matrix or its scaled version to initialize the recurrent weight matrix. We find that our solution is comparable to LSTM on our four benchmarks: two toy problems involving long-range temporal structures, a large language modeling problem and a benchmark speech recognition problem.

References in corpus (7)

• Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
• Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
• Grammar as a Foreign Language
• Learning to Execute
• Learning Longer Memory in Recurrent Neural Networks
• Addressing the Rare Word Problem in Neural Machine Translation
• Random Walk Initialization for Training Very Deep Feedforward Networks

Cited by in corpus (182)

• An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling
• Deep Facial Expression Recognition: A Survey
• Towards Large-Scale Small Object Detection: Survey and Benchmarks
• FractalNet: Ultra-Deep Neural Networks without Residuals
• Recent Advances in Recurrent Neural Networks
• SNIP: Single-shot Network Pruning based on Connection Sensitivity
• Light Gated Recurrent Units for Speech Recognition
• Artificial neural networks for neuroscientists: A primer
• Adversarial Frontier Stitching for Remote Neural Network Watermarking
• Working Memory Connections for LSTM
• Feed-Forward Networks with Attention Can Solve Some Long-Term Memory Problems
• Real-valued (Medical) Time Series Generation with Recurrent Conditional GANs
• Doctor AI: Predicting Clinical Events via Recurrent Neural Networks
• Direction-aware Spatial Context Features for Shadow Detection and Removal
• Unitary Evolution Recurrent Neural Networks
• RETAIN: An Interpretable Predictive Model for Healthcare using Reverse Time Attention Mechanism
• Bridging the Gaps Between Residual Learning, Recurrent Neural Networks and Visual Cortex
• Data Noising as Smoothing in Neural Network Language Models
• Zoneout: Regularizing RNNs by Randomly Preserving Hidden Activations
• What-and-Where to Match: Deep Spatially Multiplicative Integration Networks for Person Re-identification
• On orthogonality and learning recurrent networks with long term dependencies
• HiPPO: Recurrent Memory with Optimal Polynomial Projections
• Efficient softmax approximation for GPUs
• Independently Recurrent Neural Network (IndRNN): Building A Longer and Deeper RNN
• Continual Learning for Recurrent Neural Networks: an Empirical Evaluation
• Full-Capacity Unitary Recurrent Neural Networks
• Compressing Recurrent Neural Network with Tensor Train
• Architectural Complexity Measures of Recurrent Neural Networks
• Review: Deep Learning in Electron Microscopy
• Not All Samples Are Created Equal: Deep Learning with Importance Sampling
• GRAM: Graph-based Attention Model for Healthcare Representation Learning
• FastGRNN: A Fast, Accurate, Stable and Tiny Kilobyte Sized Gated Recurrent Neural Network
• Deep Active Inference
• Advances in All-Neural Speech Recognition
• R-Transformer: Recurrent Neural Network Enhanced Transformer
• Characterizing Driving Styles with Deep Learning
• Tunable Efficient Unitary Neural Networks (EUNN) and their application to RNNs
• Learning Longer-term Dependencies in RNNs with Auxiliary Losses
• Cheap Orthogonal Constraints in Neural Networks: A Simple Parametrization of the Orthogonal and Unitary Group
• SAC-Net: Spatial Attenuation Context for Salient Object Detection
• Noisy Activation Functions
• Overcoming the vanishing gradient problem in plain recurrent networks
• Understanding the computation of time using neural network models
• Implicit Regularization in Deep Learning
• Towards End-to-End Speech Recognition with Deep Convolutional Neural Networks
• The unreasonable effectiveness of the forget gate
• Low-Shot Learning from Imaginary Data
• What is the best RNN-cell structure to forecast each time series behavior?
• Improving performance of recurrent neural network with relu nonlinearity
• Inside-Outside Net: Detecting Objects in Context with Skip Pooling and Recurrent Neural Networks
• Recurrent Batch Normalization
• Natural Language Understanding with Distributed Representation
• Deeply-Recursive Convolutional Network for Image Super-Resolution
• How to Start Training: The Effect of Initialization and Architecture
• Lipschitz Recurrent Neural Networks
• Dual Rectified Linear Units (DReLUs): A Replacement for Tanh Activation Functions in Quasi-Recurrent Neural Networks
• DNA-Level Splice Junction Prediction using Deep Recurrent Neural Networks
• Revisiting Activation Regularization for Language RNNs
• Dirichlet Energy Constrained Learning for Deep Graph Neural Networks
• Universality and individuality in neural dynamics across large populations of recurrent networks
• Skip RNN: Learning to Skip State Updates in Recurrent Neural Networks
• Trivializations for Gradient-Based Optimization on Manifolds
• Deep Independently Recurrent Neural Network (IndRNN)
• Machine Learning on Sequential Data Using a Recurrent Weighted Average
• Dynamic Neural Turing Machine with Soft and Hard Addressing Schemes
• Provable Benefit of Orthogonal Initialization in Optimizing Deep Linear Networks
• BlackOut: Speeding up Recurrent Neural Network Language Models With Very Large Vocabularies
• A Taxonomy for Neural Memory Networks
• Physics Informed LSTM Network for Flexibility Identification in Evaporative Cooling Systems
• Improving the Gating Mechanism of Recurrent Neural Networks
• Deep Learning with a Rethinking Structure for Multi-label Classification
• Regularizing RNNs for Caption Generation by Reconstructing The Past with The Present
• A Parallel-Hierarchical Model for Machine Comprehension on Sparse Data
• FlexConv: Continuous Kernel Convolutions with Differentiable Kernel Sizes
• deepTarget: End-to-end Learning Framework for microRNA Target Prediction using Deep Recurrent Neural Networks
• Learning with hidden variables
• Wider and Deeper, Cheaper and Faster: Tensorized LSTMs for Sequence Learning
• Analyzing and Exploiting NARX Recurrent Neural Networks for Long-Term Dependencies
• QA4IE: A Question Answering based Framework for Information Extraction
• On Generalization Bounds of a Family of Recurrent Neural Networks
• Non-normal Recurrent Neural Network (nnRNN): learning long time dependencies while improving expressivity with transient dynamics
• Classification of Periodic Variable Stars with Novel Cyclic-Permutation Invariant Neural Networks
• Stabilizing Gradients for Deep Neural Networks via Efficient SVD Parameterization
• Facial Emotion Recognition using Convolutional Neural Networks
• Can recurrent neural networks warp time?
• HUBERT Untangles BERT to Improve Transfer across NLP Tasks
• Path-Normalized Optimization of Recurrent Neural Networks with ReLU Activations
• The Statistical Recurrent Unit
• Sliced Recurrent Neural Networks
• Spatial Attentive Single-Image Deraining with a High Quality Real Rain Dataset
• Coupled Oscillatory Recurrent Neural Network (coRNN): An accurate and (gradient) stable architecture for learning long time dependencies
• A Deep Learning Spatiotemporal Prediction Framework for Mobile Crowdsourced Services
• Strongly-Typed Recurrent Neural Networks
• Learning to Remember More with Less Memorization
• On the Initialization of Long Short-Term Memory Networks
• Calibration, Entropy Rates, and Memory in Language Models
• CKConv: Continuous Kernel Convolution For Sequential Data
• RRA: Recurrent Residual Attention for Sequence Learning
• Demystifying Deep Learning in Predictive Spatio-Temporal Analytics: An Information-Theoretic Framework
• Parallelizing Legendre Memory Unit Training
• UnICORNN: A recurrent model for learning very long time dependencies
• Slower is Better: Revisiting the Forgetting Mechanism in LSTM for Slower Information Decay
• Norm-preserving Orthogonal Permutation Linear Unit Activation Functions (OPLU)
• Effective and Efficient Computation with Multiple-timescale Spiking Recurrent Neural Networks
• State-Regularized Recurrent Neural Networks
• Identifying nonlinear dynamical systems with multiple time scales and long-range dependencies
• Deep Co-attention based Comparators For Relative Representation Learning in Person Re-identification
• Stability of Internal States in Recurrent Neural Networks Trained on Regular Languages
• Kronecker Recurrent Units
• A Simple LSTM model for Transition-based Dependency Parsing
• Kaleidoscope: An Efficient, Learnable Representation For All Structured Linear Maps
• Randomized Automatic Differentiation
• Cached Long Short-Term Memory Neural Networks for Document-Level Sentiment Classification
• Improved memory in recurrent neural networks with sequential non-normal dynamics
• Scalable Bayesian Learning of Recurrent Neural Networks for Language Modeling
• A Spike in Performance: Training Hybrid-Spiking Neural Networks with Quantized Activation Functions
• Tensor Decomposition for Compressing Recurrent Neural Network
• Noisy Recurrent Neural Networks
• The Recurrent Neural Tangent Kernel
• On the Variance of Unbiased Online Recurrent Optimization
• Low-rank passthrough neural networks
• Efficient LSTM Training with Eligibility Traces
• Untangling tradeoffs between recurrence and self-attention in neural networks
• MomentumRNN: Integrating Momentum into Recurrent Neural Networks
• Supervised training of spiking neural networks for robust deployment on mixed-signal neuromorphic processors
• An Empirical Study of Language CNN for Image Captioning
• 3D Depthwise Convolution: Reducing Model Parameters in 3D Vision Tasks
• A Basic Recurrent Neural Network Model
• Deep Recurrent Neural Networks for Sequential Phenotype Prediction in Genomics
• Contracting Implicit Recurrent Neural Networks: Stable Models with Improved Trainability
• How Large a Vocabulary Does Text Classification Need? A Variational Approach to Vocabulary Selection
• Recognizing Video Events with Varying Rhythms
• S++: A Fast and Deployable Secure-Computation Framework for Privacy-Preserving Neural Network Training
• Encoding-based Memory Modules for Recurrent Neural Networks
• Feedback Gradient Descent: Efficient and Stable Optimization with Orthogonality for DNNs
• R-FORCE: Robust Learning for Random Recurrent Neural Networks
• Refined Gate: A Simple and Effective Gating Mechanism for Recurrent Units
• An Adaptive Stochastic Nesterov Accelerated Quasi Newton Method for Training RNNs
• Complex Unitary Recurrent Neural Networks using Scaled Cayley Transform
• Learning Recurrent Binary/Ternary Weights
• Eigenvalue Normalized Recurrent Neural Networks for Short Term Memory
• Cortico-cerebellar networks as decoupling neural interfaces
• Learning to Adaptively Scale Recurrent Neural Networks
• The relationship between Biological and Artificial Intelligence
• On the biological plausibility of orthogonal initialisation for solving gradient instability in deep neural networks
• Effect in the spectra of eigenvalues and dynamics of RNNs trained with Excitatory-Inhibitory constraint
• Impact of facial landmark localization on facial expression recognition
• Character-level Deep Conflation for Business Data Analytics
• Multi-Task Learning for Argumentation Mining
• Zero Initialization of modified Gated Recurrent Encoder-Decoder Network for Short Term Load Forecasting
• Mapping high-performance RNNs to in-memory neuromorphic chips
• Better Long-Range Dependency By Bootstrapping A Mutual Information Regularizer
• Learning Simpler Language Models with the Differential State Framework
• Spectral Pruning for Recurrent Neural Networks
• Predicting Patient State-of-Health using Sliding Window and Recurrent Classifiers
• Statistical Parametric Speech Synthesis Using Bottleneck Representation From Sequence Auto-encoder
• Gradients are Not All You Need
• Empirical Evaluation of A New Approach to Simplifying Long Short-term Memory (LSTM)
• Bayesian Sparsification of Gated Recurrent Neural Networks
• Recurrent Neural Network from Adder's Perspective: Carry-lookahead RNN
• Memory and attention in deep learning
• Discrete Function Bases and Convolutional Neural Networks
• A Fully Trainable Network with RNN-based Pooling
• Learning Various Length Dependence by Dual Recurrent Neural Networks
• Constructing Gradient Controllable Recurrent Neural Networks Using Hamiltonian Dynamics
• Network of Recurrent Neural Networks
• PDRNN: Modular Data-driven Pedestrian Dead Reckoning on Loosely Coupled Radio- and Inertial-Signalstreams
• Target Propagation via Regularized Inversion
• Back to Square One: Superhuman Performance in Chutes and Ladders Through Deep Neural Networks and Tree Search
• Solving hybrid machine learning tasks by traversing weight space geodesics
• Thick-Net: Parallel Network Structure for Sequential Modeling
• Depth Enables Long-Term Memory for Recurrent Neural Networks
• CryptoGRU: Low Latency Privacy-Preserving Text Analysis With GRU
• RNN Training along Locally Optimal Trajectories via Frank-Wolfe Algorithm
• Multimodal Language Analysis with Recurrent Multistage Fusion
• Short-Term Memory Optimization in Recurrent Neural Networks by Autoencoder-based Initialization
• An empirical analysis of phrase-based and neural machine translation
• Deep Symbolic Representation Learning for Heterogeneous Time-series Classification
• Online Spatiotemporal Action Detection and Prediction via Causal Representations
• Relational Weight Priors in Neural Networks for Abstract Pattern Learning and Language Modelling
• Learning Longer-term Dependencies via Grouped Distributor Unit
• Graphical RNN Models