Analyzing and Exploiting NARX Recurrent Neural Networks for Long-Term Dependencies

Abstract

Recurrent neural networks (RNNs) have achieved state-of-the-art performance on many diverse tasks, from machine translation to surgical activity recognition, yet training RNNs to capture long-term dependencies remains difficult. To date, the vast majority of successful RNN architectures alleviate this problem using nearly-additive connections between states, as introduced by long short-term memory (LSTM). We take an orthogonal approach and introduce MIST RNNs, a NARX RNN architecture that allows direct connections from the very distant past. We show that MIST RNNs 1) exhibit superior vanishing-gradient properties in comparison to LSTM and previously-proposed NARX RNNs; 2) are far more efficient than previously-proposed NARX RNN architectures, requiring even fewer computations than LSTM; and 3) improve performance substantially over LSTM and Clockwork RNNs on tasks requiring very long-term dependencies.

References in corpus (2)

• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• A Simple Way to Initialize Recurrent Networks of Rectified Linear Units

Cited by in corpus (7)

• A Multi-Horizon Quantile Recurrent Forecaster
• Hierarchical Attention-Based Recurrent Highway Networks for Time Series Prediction
• Multi-variable LSTM neural network for autoregressive exogenous model
• Recurrence along Depth: Deep Convolutional Neural Networks with Recurrent Layer Aggregation
• Reinforcement Learning Controllers for Soft Robots using Learned Environments
• Real time expert system for anomaly detection of aerators based on computer vision technology and existing surveillance cameras
• Learning Longer-term Dependencies via Grouped Distributor Unit