Dilated Recurrent Neural Networks

Abstract

Learning with recurrent neural networks (RNNs) on long sequences is a notoriously difficult task. There are three major challenges: 1) complex dependencies, 2) vanishing and exploding gradients, and 3) efficient parallelization. In this paper, we introduce a simple yet effective RNN connection structure, the DilatedRNN, which simultaneously tackles all of these challenges. The proposed architecture is characterized by multi-resolution dilated recurrent skip connections and can be combined flexibly with diverse RNN cells. Moreover, the DilatedRNN reduces the number of parameters needed and enhances training efficiency significantly, while matching state-of-the-art performance (even with standard RNN cells) in tasks involving very long-term dependencies. To provide a theory-based quantification of the architecture's advantages, we introduce a memory capacity measure, the mean recurrent length, which is more suitable for RNNs with long skip connections than existing measures. We rigorously prove the advantages of the DilatedRNN over other recurrent neural architectures. The code for our method is publicly available at https://github.com/code-terminator/DilatedRNN

Accepted by NIPS 2017

Cited by in corpus (15)

• An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling
• Deep Learning for Time Series Anomaly Detection: A Survey
• Variational Neural Annealing
• The unreasonable effectiveness of the forget gate
• Unsupervised Deep Learning for IoT Time Series
• The Portiloop: a deep learning-based open science tool for closed-loop brain stimulation
• Wider and Deeper, Cheaper and Faster: Tensorized LSTMs for Sequence Learning
• Supplementing Recurrent Neural Networks with Annealing to Solve Combinatorial Optimization Problems
• A Time-Temperature Dataset for the Strawberry Cold Chain Across Multiple Shipments and Locations
• Learning to Adaptively Scale Recurrent Neural Networks
• Long Short-Term Memory with Dynamic Skip Connections
• DMIDAS: Deep Mixed Data Sampling Regression for Long Multi-Horizon Time Series Forecasting
• DRF: A Framework for High-Accuracy Autonomous Driving Vehicle Modeling
• A Hybrid Residual Dilated LSTM end Exponential Smoothing Model for Mid-Term Electric Load Forecasting
• Linked Recurrent Neural Networks