Recent Advances in Recurrent Neural Networks

Abstract

Recurrent neural networks (RNNs) are capable of learning features and long term dependencies from sequential and time-series data. The RNNs have a stack of non-linear units where at least one connection between units forms a directed cycle. A well-trained RNN can model any dynamical system; however, training RNNs is mostly plagued by issues in learning long-term dependencies. In this paper, we present a survey on RNNs and several new advances for newcomers and professionals in the field. The fundamentals and recent advances are explained and the research challenges are introduced.

arXiv commentarXiv admin note: text overlap with arXiv:1602.04335

References in corpus (28)

• Adam: A Method for Stochastic Optimization
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting
• An overview of gradient descent optimization algorithms
• WaveNet: A Generative Model for Raw Audio
• Pixel Recurrent Neural Networks
• DRAW: A Recurrent Neural Network For Image Generation
• Long Short-Term Memory Based Recurrent Neural Network Architectures for Large Vocabulary Speech Recognition
• Ask Me Anything: Dynamic Memory Networks for Natural Language Processing
• A Simple Way to Initialize Recurrent Networks of Rectified Linear Units
• Modeling Temporal Dependencies in High-Dimensional Sequences: Application to Polyphonic Music Generation and Transcription
• Grid Long Short-Term Memory
• How to Construct Deep Recurrent Neural Networks
• ReNet: A Recurrent Neural Network Based Alternative to Convolutional Networks
• Learning Longer Memory in Recurrent Neural Networks
• Describing Videos by Exploiting Temporal Structure
• Zoneout: Regularizing RNNs by Randomly Preserving Hidden Activations
• Differential Recurrent Neural Networks for Action Recognition
• Recurrent Dropout without Memory Loss
• Improved Transition-Based Parsing by Modeling Characters instead of Words with LSTMs
• Learning Natural Language Inference with LSTM
• Depth-Gated LSTM
• Multi-Dimensional Recurrent Neural Networks
• Recurrent Generative Adversarial Networks for Proximal Learning and Automated Compressive Image Recovery
• A Neural Transducer
• Semantic Object Parsing with Local-Global Long Short-Term Memory
• Regularizing RNNs by Stabilizing Activations
• Recurrent Memory Networks for Language Modeling

Cited by in corpus (37)

• Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) Network
• How to Build a Graph-Based Deep Learning Architecture in Traffic Domain: A Survey
• Recurrent Neural Networks: An Embedded Computing Perspective
• On the Convergence Rate of Training Recurrent Neural Networks
• DynNet: Physics-based neural architecture design for linear and nonlinear structural response modeling and prediction
• Deep Recurrent Neural Networks for ECG Signal Denoising
• Training Optimization for Gate-Model Quantum Neural Networks
• Survey of Dropout Methods for Deep Neural Networks
• Finite Element Network Analysis: A Machine Learning based Computational Framework for the Simulation of Physical Systems
• A Survey of Optimization Methods from a Machine Learning Perspective
• LiteHAR: Lightweight Human Activity Recognition from WiFi Signals with Random Convolution Kernels
• Recent Advances in Deep Learning for Channel Coding: A Survey
• Reservoir Computing for Macroeconomic Forecasting with Mixed Frequency Data
• Deep Sequence Modeling: Development and Applications in Asset Pricing
• Vau da muntanialas: Energy-efficient multi-die scalable acceleration of RNN inference
• Reservoir Computing meets Recurrent Kernels and Structured Transforms
• Can SGD Learn Recurrent Neural Networks with Provable Generalization?
• Prediction of the Position of External Markers Using a Recurrent Neural Network Trained With Unbiased Online Recurrent Optimization for Safe Lung Cancer Radiotherapy
• EnGN: A High-Throughput and Energy-Efficient Accelerator for Large Graph Neural Networks
• $l_2$ Induced Norm Analysis of Discrete-Time LTI Systems for Nonnegative Input Signals and Its Application to Stability Analysis of Recurrent Neural Networks
• Rethinking Recurrent Neural Networks and Other Improvements for Image Classification
• Initializing LSTM internal states via manifold learning
• seq2graph: Discovering Dynamic Dependencies from Multivariate Time Series with Multi-level Attention
• A Preliminary Study on Data Augmentation of Deep Learning for Image Classification
• Predicting Stellar Mass Accretion: An Optimized Echo State Network Approach in Time Series Modeling
• Intrusion Detection System in Smart Home Network Using Bidirectional LSTM and Convolutional Neural Networks Hybrid Model
• Ising-Dropout: A Regularization Method for Training and Compression of Deep Neural Networks
• A Mathematical Framework for Learning Probability Distributions
• Variable-Order Approach to Nonlocal Elasticity: Theoretical Formulation and Order Identification via Deep Learning Techniques
• ViPTT-Net: Video pretraining of spatio-temporal model for tuberculosis type classification from chest CT scans
• Robust Online Model Adaptation by Extended Kalman Filter with Exponential Moving Average and Dynamic Multi-Epoch Strategy
• Automated Monitoring Cropland Using Remote Sensing Data: Challenges and Opportunities for Machine Learning
• Detecting Markovianity of Quantum Processes via Recurrent Neural Networks
• Deep Convolutional Generative Adversarial Network Based Food Recognition Using Partially Labeled Data
• Cylindrical Transform: 3D Semantic Segmentation of Kidneys With Limited Annotated Images
• Algorithms and Hardness for Linear Algebra on Geometric Graphs
• Surrogate Modeling for Scalable Evaluation of Distributed Computing Systems for HEP Applications