A Critical Review of Recurrent Neural Networks for Sequence Learning

Abstract

Countless learning tasks require dealing with sequential data. Image captioning, speech synthesis, and music generation all require that a model produce outputs that are sequences. In other domains, such as time series prediction, video analysis, and musical information retrieval, a model must learn from inputs that are sequences. Interactive tasks, such as translating natural language, engaging in dialogue, and controlling a robot, often demand both capabilities. Recurrent neural networks (RNNs) are connectionist models that capture the dynamics of sequences via cycles in the network of nodes. Unlike standard feedforward neural networks, recurrent networks retain a state that can represent information from an arbitrarily long context window. Although recurrent neural networks have traditionally been difficult to train, and often contain millions of parameters, recent advances in network architectures, optimization techniques, and parallel computation have enabled successful large-scale learning with them. In recent years, systems based on long short-term memory (LSTM) and bidirectional (BRNN) architectures have demonstrated ground-breaking performance on tasks as varied as image captioning, language translation, and handwriting recognition. In this survey, we review and synthesize the research that over the past three decades first yielded and then made practical these powerful learning models. When appropriate, we reconcile conflicting notation and nomenclature. Our goal is to provide a self-contained explication of the state of the art together with a historical perspective and references to primary research.

References in corpus (6)

• Sequence to Sequence Learning with Neural Networks
• ADADELTA: An Adaptive Learning Rate Method
• On the difficulty of training Recurrent Neural Networks
• Identifying and attacking the saddle point problem in high-dimensional non-convex optimization
• Sequence to Sequence -- Video to Text
• Efficient Elastic Net Regularization for Sparse Linear Models

Cited by in corpus (229)

• Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) Network
• Data-driven discovery of coordinates and governing equations
• Applications of Deep Learning and Reinforcement Learning to Biological Data
• Deep Learning for Audio Signal Processing
• ChatGPT: Jack of all trades, master of none
• Personalizing Session-based Recommendations with Hierarchical Recurrent Neural Networks
• Learning to Diagnose with LSTM Recurrent Neural Networks
• Recurrent Neural Networks For Accurate RSSI Indoor Localization
• Deep Bidirectional and Unidirectional LSTM Recurrent Neural Network for Network-wide Traffic Speed Prediction
• Early Stage Malware Prediction Using Recurrent Neural Networks
• An overview and comparative analysis of Recurrent Neural Networks for Short Term Load Forecasting
• Graph-Based Deep Learning for Medical Diagnosis and Analysis: Past, Present and Future
• Continuous online sequence learning with an unsupervised neural network model
• Deep Learning for Human Affect Recognition: Insights and New Developments
• Anticipating cryptocurrency prices using machine learning
• Spatial-Spectral Feature Extraction via Deep ConvLSTM Neural Networks for Hyperspectral Image Classification
• Classification of Hand Movements from EEG using a Deep Attention-based LSTM Network
• Understanding and mitigating gradient pathologies in physics-informed neural networks
• Natural Language Processing Advancements By Deep Learning: A Survey
• Modeling Missing Data in Clinical Time Series with RNNs
• A Survey on Aspect-Based Sentiment Classification
• Click-Through Rate Prediction in Online Advertising: A Literature Review
• LSTM-Based Zero-Velocity Detection for Robust Inertial Navigation
• Recurrent Neural Networks: An Embedded Computing Perspective
• Identifying Solar Flare Precursors Using Time Series of SDO/HMI Images and SHARP Parameters
• Pricing options and computing implied volatilities using neural networks
• Measurement of Anomalous Diffusion Using Recurrent Neural Networks
• Sequence-based Dynamic Handwriting Analysis for Parkinson's Disease Detection with One-dimensional Convolutions and BiGRUs
• Review: Deep Learning in Electron Microscopy
• A Survey of Deep Learning Techniques for Neural Machine Translation
• Machine learning with neural networks
• Characterizing Driving Styles with Deep Learning
• On the Origin of Deep Learning
• Robust Data-Driven Zero-Velocity Detection for Foot-Mounted Inertial Navigation
• Interference Suppression Using Deep Learning: Current Approaches and Open Challenges
• Artificial Neural Networks for Photonic Applications: From Algorithms to Implementation
• Thermodynamically Consistent Machine-Learned Internal State Variable Approach for Data-Driven Modeling of Path-Dependent Materials
• Using LSTM recurrent neural networks for monitoring the LHC superconducting magnets
• Anticipating synchronization with machine learning
• Neural Memory Networks for Seizure Type Classification
• A Review of Hidden Markov Models and Recurrent Neural Networks for Event Detection and Localization in Biomedical Signals
• Predicting rice blast disease: machine learning versus process based models
• DR-RNN: A deep residual recurrent neural network for model reduction
• The unreasonable effectiveness of the forget gate
• Content and Style Aware Generation of Text-line Images for Handwriting Recognition
• Greenhouse Gas Emission Prediction on Road Network using Deep Sequence Learning
• Data-driven control of room temperature and bidirectional EV charging using deep reinforcement learning: simulations and experiments
• Training LSTM Networks with Resistive Cross-Point Devices
• Decoding EEG Brain Activity for Multi-Modal Natural Language Processing
• Development and Evaluation of Recurrent Neural Network based Models for Hourly Traffic Volume and AADT Prediction
• Scalable End-to-end Recurrent Neural Network for Variable star classification
• Upper Esophageal Sphincter Opening Segmentation with Convolutional Recurrent Neural Networks in High Resolution Cervical Auscultation
• Lightweight Neural Architecture Search for Temporal Convolutional Networks at the Edge
• Deep Air Quality Forecasting Using Hybrid Deep Learning Framework
• Transformer-based Map Matching Model with Limited Ground-Truth Data using Transfer-Learning Approach
• Artificial intelligence and machine learning applications for cultured meat
• Artificial Intelligence and Deep Learning Algorithms for Epigenetic Sequence Analysis: A Review for Epigeneticists and AI Experts
• Mobile Video Object Detection with Temporally-Aware Feature Maps
• Deep modeling of quasar variability
• Learning Hamiltonian dynamics by reservoir computer
• A Deep Learning Framework for Simulation and Defect Prediction Applied in Microelectronics
• Deep Learning For Computer Vision Tasks: A review
• Credit Card Fraud Detection using Machine Learning: A Study
• Robust Forecasting using Predictive Generalized Synchronization in Reservoir Computing
• A Survey of Machine Learning Methods and Challenges for Windows Malware Classification
• State estimation with limited sensors -- A deep learning based approach
• U(1) symmetric recurrent neural networks for quantum state reconstruction
• A Deep Learning-based Framework for Conducting Stealthy Attacks in Industrial Control Systems
• 3G structure for image caption generation
• Recurrent neural networks and transfer learning for elasto-plasticity in woven composites
• A Survey of Natural Language Generation Techniques with a Focus on Dialogue Systems - Past, Present and Future Directions
• Tutorial: Neuromorphic spiking neural networks for temporal learning
• Augmentation Scheme for Dealing with Imbalanced Network Traffic Classification Using Deep Learning
• SIRNet: Understanding Social Distancing Measures with Hybrid Neural Network Model for COVID-19 Infectious Spread
• LSTM vs. GRU vs. Bidirectional RNN for script generation
• A Review of Machine Learning Applications in Fuzzing
• Phenotyping of Clinical Time Series with LSTM Recurrent Neural Networks
• Time Series Analysis and Modeling to Forecast: a Survey
• Multi-Objective De Novo Drug Design with Conditional Graph Generative Model
• Predicting purchasing intent: Automatic Feature Learning using Recurrent Neural Networks
• Generative Concatenative Nets Jointly Learn to Write and Classify Reviews
• The Effect of Explicit Structure Encoding of Deep Neural Networks for Symbolic Music Generation
• R2N2: Residual Recurrent Neural Networks for Multivariate Time Series Forecasting
• Arcades: A deep model for adaptive decision making in voice controlled smart-home
• Dynamical Hyperspectral Unmixing with Variational Recurrent Neural Networks
• E-LSTM-D: A Deep Learning Framework for Dynamic Network Link Prediction
• Quantum-tailored machine-learning characterization of a superconducting qubit
• Multi-frame Feature Aggregation for Real-time Instrument Segmentation in Endoscopic Video
• Machine Learning Methods in Solving the Boolean Satisfiability Problem
• A Survey on Dynamic Network Embedding
• gym-gazebo2, a toolkit for reinforcement learning using ROS 2 and Gazebo
• Deep Sequence Modeling: Development and Applications in Asset Pricing
• Random Error Sampling-based Recurrent Neural Network Architecture Optimization
• Listening to Sounds of Silence for Speech Denoising
• Neural Language Generation: Formulation, Methods, and Evaluation
• The model of an anomaly detector for HiLumi LHC magnets based on Recurrent Neural Networks and adaptive quantization
• Sales forecasting using WaveNet within the framework of the Kaggle competition
• Forecasting the Spread of Covid-19 Under Control Scenarios Using LSTM and Dynamic Behavioral Models
• Neural Translation of Musical Style
• SPI: Automated Identification of Security Patches via Commits
• Beyond Graph Neural Networks with Lifted Relational Neural Networks
• Evolution of transfer learning in natural language processing
• DeepRT: deep learning for peptide retention time prediction in proteomics
• Inverse Modeling of Viscoelasticity Materials using Physics Constrained Learning
• Deep Learning in Mining Biological Data
• Learning-based Nonlinear Model Predictive Control of Articulated Soft Robots using Recurrent Neural Networks
• Generating Emotive Gaits for Virtual Agents Using Affect-Based Autoregression
• Predicting human decision making in psychological tasks with recurrent neural networks
• Progressive Operational Perceptron with Memory
• Techniques for visualizing LSTMs applied to electrocardiograms
• DeepCCI: End-to-end Deep Learning for Chemical-Chemical Interaction Prediction
• Supplementing Recurrent Neural Networks with Annealing to Solve Combinatorial Optimization Problems
• Machine learning classification of non-Markovian noise disturbing quantum dynamics
• A CNN-LSTM Quantifier for Single Access Point CSI Indoor Localization
• PharmMT: A Neural Machine Translation Approach to Simplify Prescription Directions
• FATA-Trans: Field And Time-Aware Transformer for Sequential Tabular Data
• Learning Deep Representations for Semantic Image Parsing: a Comprehensive Overview
• Pruning In Time (PIT): A Lightweight Network Architecture Optimizer for Temporal Convolutional Networks
• DeepCover: Advancing RNN Test Coverage and Online Error Prediction using State Machine Extraction
• A Deep Prediction Network for Understanding Advertiser Intent and Satisfaction
• Beam Search with Bidirectional Strategies for Neural Response Generation
• Low-Cost Recurrent Neural Network Expected Performance Evaluation
• Effectiveness of LSTMs in Predicting Congestive Heart Failure Onset
• Arabic Multi-Dialect Segmentation: bi-LSTM-CRF vs. SVM
• Training Input-Output Recurrent Neural Networks through Spectral Methods
• Recurrent Localization Networks applied to the Lippmann-Schwinger Equation
• How deep learning works --The geometry of deep learning
• Debiasing the Cloze Task in Sequential Recommendation with Bidirectional Transformers
• client2vec: Towards Systematic Baselines for Banking Applications
• Improving Context Modeling in Neural Topic Segmentation
• RC-RNN: Reconfigurable Cache Architecture for Storage Systems Using Recurrent Neural Networks
• Audio Captioning using Gated Recurrent Units
• Incremental Sequence Learning
• Multivariate, Multistep Forecasting, Reconstruction and Feature Selection of Ocean Waves via Recurrent and Sequence-to-Sequence Networks
• Automatic Model Selection for Neural Networks
• Concept-Oriented Deep Learning: Generative Concept Representations
• Adding Attentiveness to the Neurons in Recurrent Neural Networks
• Robust State Estimation from Partial Out-Core Measurements with Shallow Recurrent Decoder for Nuclear Reactors
• Robust Named Entity Recognition in Idiosyncratic Domains
• Emotional Video to Audio Transformation Using Deep Recurrent Neural Networks and a Neuro-Fuzzy System
• Deep Learning and LLM-based Methods Applied to Stellar Lightcurve Classification
• Seglearn: A Python Package for Learning Sequences and Time Series
• Sparseout: Controlling Sparsity in Deep Networks
• Rethinking Recurrent Neural Networks and Other Improvements for Image Classification
• Recurrent Neural Networks for anomaly detection in the Post-Mortem time series of LHC superconducting magnets
• Connecting Software Metrics across Versions to Predict Defects
• Dance Dance Convolution
• Neural Dynamics on Complex Networks
• Hidden Markov Chains, Entropic Forward-Backward, and Part-Of-Speech Tagging
• Recurrent neural network wave functions for Rydberg atom arrays on kagome lattice
• Deep Learning-aided Application Scheduler for Vehicular Safety Communication
• Deep Learning in Computer-Aided Diagnosis and Treatment of Tumors: A Survey
• Multi-Level Recurrent Residual Networks for Action Recognition
• Survey of ETA prediction methods in public transport networks
• The Myths of Our Time: Fake News
• Long Short-Term Attention
• Notes on Deep Learning for NLP
• Early Predictions for Medical Crowdfunding: A Deep Learning Approach Using Diverse Inputs
• Deep Trustworthy Knowledge Tracing
• Highly Fast Text Segmentation With Pairwise Markov Chains
• Fusion Recurrent Neural Network
• Deep Probabilistic Time Series Forecasting using Augmented Recurrent Input for Dynamic Systems
• Efficient recurrent neural network methods for anomalously diffusing single particle short and noisy trajectories
• Indoor thermal comfort management: A Bayesian machine-learning approach to data denoising and dynamics prediction of HVAC systems
• Candidate Fusion: Integrating Language Modelling into a Sequence-to-Sequence Handwritten Word Recognition Architecture
• Tensor-Train Long Short-Term Memory for Monaural Speech Enhancement
• Sequence-Aware Recommender Systems
• Parkinson's Disease Diagnosis Using Deep Learning
• Partial Differential Equations is All You Need for Generating Neural Architectures -- A Theory for Physical Artificial Intelligence Systems
• Taxi Demand-Supply Forecasting: Impact of Spatial Partitioning on the Performance of Neural Networks
• Forecasting Using Reservoir Computing: The Role of Generalized Synchronization
• A Deep Learning Approach to the Inversion of Borehole Resistivity Measurements
• Ground Reaction Force Estimation via Time-aware Knowledge Distillation
• Modeling Musical Taste Evolution with Recurrent Neural Networks
• A Dynamically Controlled Recurrent Neural Network for Modeling Dynamical Systems
• COLREG-Compliant Collision Avoidance for Unmanned Surface Vehicle using Deep Reinforcement Learning
• Weighted graphlets and deep neural networks for protein structure classification
• An In-depth Summary of Recent Artificial Intelligence Applications in Drug Design
• Modeling the spatio-temporal dynamics of land use change with recurrent neural networks
• Physics-informed Transformers for Electronic Quantum States
• General Backpropagation Algorithm for Training Second-order Neural Networks
• RecNets: Channel-wise Recurrent Convolutional Neural Networks
• Explore Recurrent Neural Network for PUE Attack Detection in Practical CRN Models
• DETECT: Deep Trajectory Clustering for Mobility-Behavior Analysis
• Deep Reinforcement Learning From Raw Pixels in Doom
• Session-Based Recommender Systems for Action Selection in GUI Test Generation
• A Deep Learning Approach for Similar Languages, Varieties and Dialects
• DeepDPM: Dynamic Population Mapping via Deep Neural Network
• Evolvable Psychology Informed Neural Network for Memory Behavior Modeling
• Rumour Detection via News Propagation Dynamics and User Representation Learning
• Statistical Downscaling of Temperature Distributions from the Synoptic Scale to the Mesoscale Using Deep Convolutional Neural Networks
• Forecasting solar radiation during dust storms using deep learning
• Recurrent Control Nets for Deep Reinforcement Learning
• Recovering discrete delayed fractional equations from trajectories
• OadTR: Online Action Detection with Transformers
• AI-Based Detection, Classification and Prediction/Prognosis in Medical Imaging: Towards Radiophenomics
• Learning Nonlinear Waves in Plasmon-induced Transparency
• DeepAg: Deep Learning Approach for Measuring the Effects of Outlier Events on Agricultural Production and Policy
• A GAN-based Approach for Mitigating Inference Attacks in Smart Home Environment
• Construction of a far ultraviolet all sky map from an incomplete survey: Application of a deep learning algorithm
• RNN-based Online Learning: An Efficient First-Order Optimization Algorithm with a Convergence Guarantee
• Differentiate Everything with a Reversible Embeded Domain-Specific Language
• Multimodal Data Fusion based on the Global Workspace Theory
• Exploiting Review Neighbors for Contextualized Helpfulness Prediction
• SeER: An Explainable Deep Learning MIDI-based Hybrid Song Recommender System
• TIES: Temporal Interaction Embeddings For Enhancing Social Media Integrity At Facebook
• A Multilingual Encoding Method for Text Classification and Dialect Identification Using Convolutional Neural Network
• Addressing Two Problems in Deep Knowledge Tracing via Prediction-Consistent Regularization
• Low-cost Autonomous Navigation System Based on Optical Flow Classification
• Sleep Stage Classification Based on Multi-level Feature Learning and Recurrent Neural Networks via Wearable Device
• Machine Learning for Protein Function
• A Conceptual Development of Quench Prediction App build on LSTM and ELQA framework
• Linked Recurrent Neural Networks
• Reducing state updates via Gaussian-gated LSTMs
• Brain decoding from functional MRI using long short-term memory recurrent neural networks
• Time-Sequence Channel Inference for Beam Alignment in Vehicular Networks
• A Survey of Brain Inspired Technologies for Engineering
• A Deep Learning Approach for Automotive Radar Interference Mitigation
• BGADAM: Boosting based Genetic-Evolutionary ADAM for Neural Network Optimization
• Online Dynamic Network Embedding
• A Bayesian Long Short-Term Memory Model for Value at Risk and Expected Shortfall Joint Forecasting
• Characterization and recognition of handwritten digits using Julia
• geoGAT: Graph Model Based on Attention Mechanism for Geographic Text Classification
• Low-Dimensional Manifolds Support Multiplexed Integrations in Recurrent Neural Networks
• Predictive process mining by network of classifiers and clusterers: the PEDF model
• Identifying the fragment structure of the organic compounds by deeply learning the original NMR data
• Predicting Drug Responses by Propagating Interactions through Text-Enhanced Drug-Gene Networks
• Introducing a Generative Adversarial Network Model for Lagrangian Trajectory Simulation
• Network Learning with Local Propagation