On Multiplicative Integration with Recurrent Neural Networks

Abstract

We introduce a general and simple structural design called Multiplicative Integration (MI) to improve recurrent neural networks (RNNs). MI changes the way in which information from difference sources flows and is integrated in the computational building block of an RNN, while introducing almost no extra parameters. The new structure can be easily embedded into many popular RNN models, including LSTMs and GRUs. We empirically analyze its learning behaviour and conduct evaluations on several tasks using different RNN models. Our experimental results demonstrate that Multiplicative Integration can provide a substantial performance boost over many of the existing RNN models.

arXiv comment10 pages, 2 figures; To appear in NIPS2016

References in corpus (8)

• Adam: A Method for Stochastic Optimization
• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• Grid Long Short-Term Memory
• EESEN: End-to-End Speech Recognition using Deep RNN Models and WFST-based Decoding
• First-Pass Large Vocabulary Continuous Speech Recognition using Bi-Directional Recurrent DNNs
• Architectural Complexity Measures of Recurrent Neural Networks
• Regularization and nonlinearities for neural language models: when are they needed?
• Regularizing RNNs by Stabilizing Activations

Cited by in corpus (26)

• An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling
• Hierarchical Multiscale Recurrent Neural Networks
• Stochastic Neural Networks for Hierarchical Reinforcement Learning
• What-and-Where to Match: Deep Spatially Multiplicative Integration Networks for Person Re-identification
• Adversarial Video Generation on Complex Datasets
• A Survey of Deep Learning Techniques for Neural Machine Translation
• Learning Longer-term Dependencies in RNNs with Auxiliary Losses
• Multiplicative LSTM for sequence modelling
• Dynamic Evaluation of Neural Sequence Models
• Learning to Create and Reuse Words in Open-Vocabulary Neural Language Modeling
• Neural Language Generation: Formulation, Methods, and Evaluation
• Neural Language Modeling by Jointly Learning Syntax and Lexicon
• Semantic Compositional Networks for Visual Captioning
• Verification of Recurrent Neural Networks Through Rule Extraction
• A Comparative Study of Rule Extraction for Recurrent Neural Networks
• Training Language Models Using Target-Propagation
• Breaking the Activation Function Bottleneck through Adaptive Parameterization
• Surprisal-Driven Zoneout
• Weighted Sigmoid Gate Unit for an Activation Function of Deep Neural Network
• Multiscale sequence modeling with a learned dictionary
• Deep Variational Inference Without Pixel-Wise Reconstruction
• Refined Gate: A Simple and Effective Gating Mechanism for Recurrent Units
• On the Long-Term Memory of Deep Recurrent Networks
• Multi-Zone Unit for Recurrent Neural Networks
• Learning Simpler Language Models with the Differential State Framework
• Recognizing Long Grammatical Sequences Using Recurrent Networks Augmented With An External Differentiable Stack