Recurrent Additive Networks

Abstract

We introduce recurrent additive networks (RANs), a new gated RNN which is distinguished by the use of purely additive latent state updates. At every time step, the new state is computed as a gated component-wise sum of the input and the previous state, without any of the non-linearities commonly used in RNN transition dynamics. We formally show that RAN states are weighted sums of the input vectors, and that the gates only contribute to computing the weights of these sums. Despite this relatively simple functional form, experiments demonstrate that RANs perform on par with LSTMs on benchmark language modeling problems. This result shows that many of the non-linear computations in LSTMs and related networks are not essential, at least for the problems we consider, and suggests that the gates are doing more of the computational work than previously understood.

References in corpus (4)

• Neural Architecture Search with Reinforcement Learning
• Recurrent Neural Network Regularization
• Exploring the Limits of Language Modeling
• Multiplicative LSTM for sequence modelling

Cited by in corpus (10)

• Deriving Neural Architectures from Sequence and Graph Kernels
• Self-Attentive Residual Decoder for Neural Machine Translation
• Echo: Compiler-based GPU Memory Footprint Reduction for LSTM RNN Training
• Bridging the Knowledge Gap: Enhancing Question Answering with World and Domain Knowledge
• SoPa: Bridging CNNs, RNNs, and Weighted Finite-State Machines
• Simplifying Neural Machine Translation with Addition-Subtraction Twin-Gated Recurrent Networks
• A Lightweight Recurrent Network for Sequence Modeling
• Non-Projective Dependency Parsing via Latent Heads Representation (LHR)
• Rational Recurrences
• An Augmented Transformer Architecture for Natural Language Generation Tasks