Deep Independently Recurrent Neural Network (IndRNN)

Abstract

Recurrent neural networks (RNNs) are known to be difficult to train due to the gradient vanishing and exploding problems and thus difficult to learn long-term patterns and construct deep networks. To address these problems, this paper proposes a new type of RNNs with the recurrent connection formulated as Hadamard product, referred to as independently recurrent neural network (IndRNN), where neurons in the same layer are independent of each other and connected across layers. Due to the better behaved gradient backpropagation, IndRNN with regulated recurrent weights effectively addresses the gradient vanishing and exploding problems and thus long-term dependencies can be learned. Moreover, an IndRNN can work with non-saturated activation functions such as ReLU (rectified linear unit) and be still trained robustly. Different deeper IndRNN architectures, including the basic stacked IndRNN, residual IndRNN and densely connected IndRNN, have been investigated, all of which can be much deeper than the existing RNNs. Furthermore, IndRNN reduces the computation at each time step and can be over 10 times faster than the commonly used Long short-term memory (LSTM). Experimental results have shown that the proposed IndRNN is able to process very long sequences and construct very deep networks. Better performance has been achieved on various tasks with IndRNNs compared with the traditional RNN, LSTM and the popular Transformer.

arXiv commentExtension of the CVPR2018 paper "Independently Recurrent Neural Network (IndRNN): Building A Longer and Deeper RNN", with significant improvements as described in the end of Section I. arXiv admin note: text overlap with arXiv:1803.04831

References in corpus (18)

• LSTM: A Search Space Odyssey
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Neural Architecture Search with Reinforcement Learning
• On the difficulty of training Recurrent Neural Networks
• NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding
• Character-Aware Neural Language Models
• A Simple Way to Initialize Recurrent Networks of Rectified Linear Units
• An End-to-End Spatio-Temporal Attention Model for Human Action Recognition from Skeleton Data
• Gated Feedback Recurrent Neural Networks
• SkeleMotion: A New Representation of Skeleton Joint Sequences Based on Motion Information for 3D Action Recognition
• Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modeling
• Part-based Graph Convolutional Network for Action Recognition
• Improving Neural Language Models with a Continuous Cache
• R-Transformer: Recurrent Neural Network Enhanced Transformer
• Capacity and Trainability in Recurrent Neural Networks
• Pose-conditioned Spatio-Temporal Attention for Human Action Recognition
• Improving performance of recurrent neural network with relu nonlinearity
• Learning Long Term Dependencies via Fourier Recurrent Units

Cited by in corpus (8)

• Human Action Recognition from Various Data Modalities: A Review
• UnICORNN: A recurrent model for learning very long time dependencies
• Effective and Efficient Computation with Multiple-timescale Spiking Recurrent Neural Networks
• A Two-stream Neural Network for Pose-based Hand Gesture Recognition
• DeepSITH: Efficient Learning via Decomposition of What and When Across Time Scales
• RIANN -- A Robust Neural Network Outperforms Attitude Estimation Filters
• A Chain Graph Interpretation of Real-World Neural Networks
• A Framework of Combining Short-Term Spatial/Frequency Feature Extraction and Long-Term IndRNN for Activity Recognition