State-Regularized Recurrent Neural Networks

Abstract

Recurrent neural networks are a widely used class of neural architectures. They have, however, two shortcomings. First, it is difficult to understand what exactly they learn. Second, they tend to work poorly on sequences requiring long-term memorization, despite having this capacity in principle. We aim to address both shortcomings with a class of recurrent networks that use a stochastic state transition mechanism between cell applications. This mechanism, which we term state-regularization, makes RNNs transition between a finite set of learnable states. We evaluate state-regularized RNNs on (1) regular languages for the purpose of automata extraction; (2) nonregular languages such as balanced parentheses, palindromes, and the copy task where external memory is required; and (3) real-word sequence learning tasks for sentiment analysis, visual object recognition, and language modeling. We show that state-regularization (a) simplifies the extraction of finite state automata modeling an RNN's state transition dynamics; (b) forces RNNs to operate more like automata with external memory and less like finite state machines; (c) makes RNNs have better interpretability and explainability.

to appear at ICML2019, 20 pages

References in corpus (14)

• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling
• Recurrent Neural Network Regularization
• A Simple Way to Initialize Recurrent Networks of Rectified Linear Units
• Semi-supervised Sequence Learning
• Learning to Execute
• Learning Stochastic Recurrent Networks
• Learning to Transduce with Unbounded Memory
• Associative Long Short-Term Memory
• Higher Order Recurrent Neural Networks
• Machine Learning on Sequential Data Using a Recurrent Weighted Average
• Noisin: Unbiased Regularization for Recurrent Neural Networks
• A Comparative Study of Rule Extraction for Recurrent Neural Networks

Cited by in corpus (6)

• Learning Long-Term Dependencies in Irregularly-Sampled Time Series
• Stability of Internal States in Recurrent Neural Networks Trained on Regular Languages
• Handling Long-Tail Queries with Slice-Aware Conversational Systems
• Distance and Equivalence between Finite State Machines and Recurrent Neural Networks: Computational results
• On the Memory Mechanism of Tensor-Power Recurrent Models
• Aging Memories Generate More Fluent Dialogue Responses with Memory Augmented Neural Networks