A recurrent neural network without chaos

Abstract

We introduce an exceptionally simple gated recurrent neural network (RNN) that achieves performance comparable to well-known gated architectures, such as LSTMs and GRUs, on the word-level language modeling task. We prove that our model has simple, predicable and non-chaotic dynamics. This stands in stark contrast to more standard gated architectures, whose underlying dynamical systems exhibit chaotic behavior.

References in corpus (3)

• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• Recurrent Neural Network Regularization
• Learning Longer Memory in Recurrent Neural Networks

Cited by in corpus (12)

• Stable Recurrent Models
• NAIS-Net: Stable Deep Networks from Non-Autonomous Differential Equations
• Forecasting Sequential Data using Consistent Koopman Autoencoders
• MinimalRNN: Toward More Interpretable and Trainable Recurrent Neural Networks
• Coupled Oscillatory Recurrent Neural Network (coRNN): An accurate and (gradient) stable architecture for learning long time dependencies
• Top-K Off-Policy Correction for a REINFORCE Recommender System
• UnICORNN: A recurrent model for learning very long time dependencies
• Privacy-Aware Recommender Systems Challenge on Twitter's Home Timeline
• MomentumRNN: Integrating Momentum into Recurrent Neural Networks
• A non-autonomous equation discovery method for time signal classification
• R-FORCE: Robust Learning for Random Recurrent Neural Networks
• Iterative evaluation of LSTM cells