Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) Network

Abstract

Because of their effectiveness in broad practical applications, LSTM networks have received a wealth of coverage in scientific journals, technical blogs, and implementation guides. However, in most articles, the inference formulas for the LSTM network and its parent, RNN, are stated axiomatically, while the training formulas are omitted altogether. In addition, the technique of "unrolling" an RNN is routinely presented without justification throughout the literature. The goal of this paper is to explain the essential RNN and LSTM fundamentals in a single document. Drawing from concepts in signal processing, we formally derive the canonical RNN formulation from differential equations. We then propose and prove a precise statement, which yields the RNN unrolling technique. We also review the difficulties with training the standard RNN and address them by transforming the RNN into the "Vanilla LSTM" network through a series of logical arguments. We provide all equations pertaining to the LSTM system together with detailed descriptions of its constituent entities. Albeit unconventional, our choice of notation and the method for presenting the LSTM system emphasizes ease of understanding. As part of the analysis, we identify new opportunities to enrich the LSTM system and incorporate these extensions into the Vanilla LSTM network, producing the most general LSTM variant to date. The target reader has already been exposed to RNNs and LSTM networks through numerous available resources and is open to an alternative pedagogical approach. A Machine Learning practitioner seeking guidance for implementing our new augmented LSTM model in software for experimentation and research will find the insights and derivations in this tutorial valuable as well.

43 pages, 10 figures, 78 references

References in corpus (11)

• Sequence to Sequence Learning with Neural Networks
• Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) Network
• On the difficulty of training Recurrent Neural Networks
• Recurrent Neural Network Regularization
• Recent Advances in Recurrent Neural Networks
• Neural Ordinary Differential Equations
• Latent ODEs for Irregularly-Sampled Time Series
• Beyond Finite Layer Neural Networks: Bridging Deep Architectures and Numerical Differential Equations
• AntisymmetricRNN: A Dynamical System View on Recurrent Neural Networks
• Reversible Architectures for Arbitrarily Deep Residual Neural Networks
• NAIS-Net: Stable Deep Networks from Non-Autonomous Differential Equations

Cited by in corpus (131)

• Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) Network
• Machine Learning in Aerodynamic Shape Optimization
• Transformers in Time-series Analysis: A Tutorial
• COVID-19 Cough Classification using Machine Learning and Global Smartphone Recordings
• Intelligent Computing: The Latest Advances, Challenges and Future
• Multimodal Co-learning: Challenges, Applications with Datasets, Recent Advances and Future Directions
• Quantum Machine Learning for Chemistry and Physics
• A deep learning approach to predict significant wave height using long short-term memory
• Machine Learning on Neutron and X-Ray Scattering
• Review: Deep Learning in Electron Microscopy
• Time series compression: a survey
• Search for heavy diboson resonances in semileptonic final states in $pp$ collisions at $\sqrt{s}=13$ TeV with the ATLAS detector
• Deep learning for reconstructing protein structures from cryo-EM density maps: recent advances and future directions
• The Evolution of Distributed Systems for Graph Neural Networks and their Origin in Graph Processing and Deep Learning: A Survey
• Learning temporal data with variational quantum recurrent neural network
• Learning Long-Term Dependencies in Irregularly-Sampled Time Series
• Strong gravitational lensing and microlensing of supernovae
• Neurosymbolic AI for Reasoning over Knowledge Graphs: A Survey
• Perspective on unconventional computing using magnetic skyrmions
• LGN-CNN: a biologically inspired CNN architecture
• Natural Language Interfaces to Data
• A comparative study of different machine learning methods for dissipative quantum dynamics
• Short-term traffic prediction using physics-aware neural networks
• Exploring ChatGPT for Next-generation Information Retrieval: Opportunities and Challenges
• Neural-network learning of SPOD latent dynamics
• Artificial Intelligence in Materials Science and Engineering: Current Landscape, Key Challenges, and Future Trajectorie
• Automatic non-invasive Cough Detection based on Accelerometer and Audio Signals
• Learnable Fourier Features for Multi-Dimensional Spatial Positional Encoding
• Video Action Understanding
• Predicting Critical Transitions in Multiscale Dynamical Systems Using Reservoir Computing
• Earth System Data Cubes: Avenues for advancing Earth system research
• A Privacy Preserving System for Movie Recommendations Using Federated Learning
• Knowledge Graph-Enabled Text-Based Automatic Personality Prediction
• HOLISMOKES -- VII. Time-delay measurement of strongly lensed Type Ia supernovae using machine learning
• SNIascore: Deep Learning Classification of Low-Resolution Supernova Spectra
• Evaluation of Time Series Forecasting Models for Estimation of PM2.5 Levels in Air
• Artificial Intelligence-based algorithms in medical image scan seg-mentation and intelligent visual-content generation -- a concise overview
• Generating Similarity Map for COVID-19 Transmission Dynamics with Topological Autoencoder
• Real-Time Machine-Learning-Based Optimization Using Input Convex Long Short-Term Memory Network
• Transfer-Recursive-Ensemble Learning for Multi-Day COVID-19 Prediction in India using Recurrent Neural Networks
• Learning Vehicle Trajectory Uncertainty
• Efficient Generator of Mathematical Expressions for Symbolic Regression
• Recent Advances in Named Entity Recognition: A Comprehensive Survey and Comparative Study
• Real-Time Measurement-Driven Reinforcement Learning Control Approach for Uncertain Nonlinear Systems
• Rapid training of quantum recurrent neural networks
• DeTiME: Diffusion-Enhanced Topic Modeling using Encoder-decoder based LLM
• A Temporal Type-2 Fuzzy System for Time-dependent Explainable Artificial Intelligence
• Artificial neural networks ensemble methodology to predict significant wave height
• Seismic-phase detection using multiple deep learning models for global and local representations of waveforms
• Model-assisted deep learning of rare extreme events from partial observations
• Point Neuron Learning: A New Physics-Informed Neural Network Architecture
• Identifying quenched jets in heavy ion collisions with machine learning
• FedDCT: Federated Learning of Large Convolutional Neural Networks on Resource Constrained Devices using Divide and Collaborative Training
• A tensor network discriminator architecture for classification of quantum data on quantum computers
• Deep Learning for Reduced Order Modelling and Efficient Temporal Evolution of Fluid Simulations
• CapillaryNet: An Automated System to Quantify Skin Capillary Density and Red Blood Cell Velocity from Handheld Vital Microscopy
• Leveraging Large Language Models and Machine Learning for Smart Contract Vulnerability Detection
• Designing deep neural networks for driver intention recognition
• Sequence-to-Sequence Forecasting-aided State Estimation for Power Systems
• Deep reinforcement learning for preparation of thermal and prethermal quantum states
• Forecasting steam mass flow in power plants using the parallel hybrid network
• Recurrent Spectral Network (RSN): shaping the basin of attraction of a discrete map to reach automated classification
• Bridging the Gap: Machine Learning to Resolve Improperly Modeled Dynamics
• Do Language Models Understand Time?
• Effects of data time lag in a decision-making system using machine learning for pork price prediction
• A Numerical Proof of Shell Model Turbulence Closure
• Machine-learning Kohn-Sham potential from dynamics in time-dependent Kohn-Sham systems
• Hamiltonian learning from time dynamics using variational algorithms
• Self-Supervised Learning for Text Recognition: A Critical Survey
• Tab2vox: CNN-Based Multivariate Multilevel Demand Forecasting Framework by Tabular-To-Voxel Image Conversion
• Formal Methods for Autonomous Systems
• Differentiable programming and its applications to dynamical systems
• Understanding Recurrent Neural Networks Using Nonequilibrium Response Theory
• HOLISMOKES -- XII. Time-delay Measurements of Strongly Lensed Type Ia Supernovae using a Long Short-Term Memory Network
• On the Curse of Memory in Recurrent Neural Networks: Approximation and Optimization Analysis
• Learning Coordination Policies over Heterogeneous Graphs for Human-Robot Teams via Recurrent Neural Schedule Propagation
• Predicting Short Term Energy Demand in Smart Grid: A Deep Learning Approach for Integrating Renewable Energy Sources in Line with SDGs 7, 9, and 13
• The Panacea Threat Intelligence and Active Defense Platform
• Multi-stage Progressive Compression of Conformer Transducer for On-device Speech Recognition
• Effectiveness and predictability of in-network storage cache for scientific workflows
• Robust Continuous-Time Beam Tracking with Liquid Neural Network
• COVIDHealth: A Benchmark Twitter Dataset and Machine Learning based Web Application for Classifying COVID-19 Discussions
• Liouville Space Neural Network Representation of Density Matrices
• Using Transformers to Provide Teachers with Personalized Feedback on their Classroom Discourse: The TalkMoves Application
• A multivariate water quality parameter prediction model using recurrent neural network
• GNSS Measurement-Based Context Recognition for Vehicle Navigation using Gated Recurrent Unit
• Access Trends of In-network Cache for Scientific Data
• A Comprehensive Survey on Deep Learning Solutions for 3D Flood Mapping
• SPformer: A Transformer Based DRL Decision Making Method for Connected Automated Vehicles
• Artificial intelligence for improved fitting of trajectories of elementary particles in inhomogeneous dense materials immersed in a magnetic field
• Classification of Traffic Using Neural Networks by Rejecting: a Novel Approach in Classifying VPN Traffic
• Deep Learning for Crime Forecasting: The Role of Mobility at Fine-grained Spatiotemporal Scales
• Deep Insights into Cognitive Decline: A Survey of Leveraging Non-Intrusive Modalities with Deep Learning Techniques
• Cluster Counting Algorithm for the CEPC Drift Chamber using LSTM and DGCNN
• Improved Actor Relation Graph based Group Activity Recognition
• ASL Recognition with Metric-Learning based Lightweight Network
• Classification of integers based on residue classes via modern deep learning algorithms
• Understanding Information Spreading Mechanisms During COVID-19 Pandemic by Analyzing the Impact of Tweet Text and User Features for Retweet Prediction
• Fluctuation based interpretable analysis scheme for quantum many-body snapshots
• Dynamic Gesture Recognition
• Consumer Transactions Simulation through Generative Adversarial Networks
• Draw your Neural Networks
• Learning Representation over Dynamic Graph using Aggregation-Diffusion Mechanism
• A Comprehensive Review on Summarizing Financial News Using Deep Learning
• Cross-platform Learning-based Fault Tolerant Surfacing Controller for Underwater Robots
• Hydroelectric Generation Forecasting with Long Short Term Memory (LSTM) Based Deep Learning Model for Turkey
• Automated Search for Configurations of Deep Neural Network Architectures
• An Unsupervised Deep Explainable AI Framework for Localization of Concurrent Replay Attacks in Nuclear Reactor Signals
• PKSpell: Data-Driven Pitch Spelling and Key Signature Estimation
• Extracting thin film structures of energy materials using transformers
• Machine Learning for Real-Time Processing of ATLAS Liquid Argon Calorimeter Signals with FPGAs
• Non-invasive hemodynamic analysis for aortic regurgitation using computational fluid dynamics and deep learning
• Accelerating Private Large Transformers Inference through Fine-grained Collaborative Computation
• Advancing Talking Head Generation: A Comprehensive Survey of Multi-Modal Methodologies, Datasets, Evaluation Metrics, and Loss Functions
• Integration of LSTM Networks in Random Forest Algorithms for Stock Market Trading Predictions
• Kernel Corrector LSTM
• When is Enough Enough? "Just Enough" Decision Making with Recurrent Neural Networks for Radio Frequency Machine Learning
• Human Activity Recognition using Deep Learning Models on Smartphones and Smartwatches Sensor Data
• Detection of Lying Electrical Vehicles in Charging Coordination Application Using Deep Learning
• Speaker-conditioned Target Speaker Extraction based on Customized LSTM Cells
• Neural Ordinary Differential Equations for Learning and Extrapolating System Dynamics Across Bifurcations
• Development of COVID-19 Booster Vaccine Policy by Microsimulation and Q-learning
• Using Deep Learning Sequence Models to Identify SARS-CoV-2 Divergence
• IIITT@Dravidian-CodeMix-FIRE2021: Transliterate or translate? Sentiment analysis of code-mixed text in Dravidian languages
• LaNet: Real-time Lane Identification by Learning Road SurfaceCharacteristics from Accelerometer Data
• Accelerometer-based Bed Occupancy Detection for Automatic, Non-invasive Long-term Cough Monitoring
• RISE-iEEG: Robust to Inter-Subject Electrodes Implantation Variability iEEG Classifier
• UIT-HWDB: Using Transferring Method to Construct A Novel Benchmark for Evaluating Unconstrained Handwriting Image Recognition in Vietnamese
• Smart and Secure CAV Networks Empowered by AI-Enabled Blockchain: The Next Frontier for Intelligent Safe Driving Assessment
• Coarse-to-Fine Multi-Resolution Diffusion Models for Trajectory Generation in Urban Systems
• Topic Classification on Spoken Documents Using Deep Acoustic and Linguistic Features