Phased LSTM: Accelerating Recurrent Network Training for Long or Event-based Sequences

Abstract

Recurrent Neural Networks (RNNs) have become the state-of-the-art choice for extracting patterns from temporal sequences. However, current RNN models are ill-suited to process irregularly sampled data triggered by events generated in continuous time by sensors or other neurons. Such data can occur, for example, when the input comes from novel event-driven artificial sensors that generate sparse, asynchronous streams of events or from multiple conventional sensors with different update intervals. In this work, we introduce the Phased LSTM model, which extends the LSTM unit by adding a new time gate. This gate is controlled by a parametrized oscillation with a frequency range that produces updates of the memory cell only during a small percentage of the cycle. Even with the sparse updates imposed by the oscillation, the Phased LSTM network achieves faster convergence than regular LSTMs on tasks which require learning of long sequences. The model naturally integrates inputs from sensors of arbitrary sampling rates, thereby opening new areas of investigation for processing asynchronous sensory events that carry timing information. It also greatly improves the performance of LSTMs in standard RNN applications, and does so with an order-of-magnitude fewer computes at runtime.

Selected for an oral presentation at NIPS, 2016

References in corpus (4)

• Adam: A Method for Stochastic Optimization
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Recurrent Dropout without Memory Loss
• Lipreading with Long Short-Term Memory

Cited by in corpus (28)

• Spatio-Temporal Backpropagation for Training High-performance Spiking Neural Networks
• Graph-based Spatial-temporal Feature Learning for Neuromorphic Vision Sensing
• Autoregressive Convolutional Neural Networks for Asynchronous Time Series
• Is Neuromorphic MNIST neuromorphic? Analyzing the discriminative power of neuromorphic datasets in the time domain
• The unreasonable effectiveness of the forget gate
• What is the best RNN-cell structure to forecast each time series behavior?
• Hybrid Macro/Micro Level Backpropagation for Training Deep Spiking Neural Networks
• Skip RNN: Learning to Skip State Updates in Recurrent Neural Networks
• Discovering Signals from Web Sources to Predict Cyber Attacks
• Recover Missing Sensor Data with Iterative Imputing Network
• Graph-Guided Network for Irregularly Sampled Multivariate Time Series
• FCN-rLSTM: Deep Spatio-Temporal Neural Networks for Vehicle Counting in City Cameras
• Episodic memory governs choices: An RNN-based reinforcement learning model for decision-making task
• Bayesian LSTMs in medicine
• A Probabilistic Autoencoder for Type Ia Supernovae Spectral Time Series
• Scheduling Computation Graphs of Deep Learning Models on Manycore CPUs
• Forecasting Individualized Disease Trajectories using Interpretable Deep Learning
• Graph-Based Object Classification for Neuromorphic Vision Sensing
• Real-time Classification from Short Event-Camera Streams using Input-filtering Neural ODEs
• Self Attention with Temporal Prior: Can We Learn More from Arrow of Time?
• There is no Artificial General Intelligence
• Attention Mechanisms for Object Recognition with Event-Based Cameras
• Multi-task Learning for Financial Forecasting
• Linked Recurrent Neural Networks
• Reducing state updates via Gaussian-gated LSTMs
• Continual Prediction of Notification Attendance with Classical and Deep Network Approaches
• HeunNet: Extending ResNet using Heun's Methods
• Analysis of memory in LSTM-RNNs for source separation