Adaptive Recurrent Neural Network Based on Mixture Layer

Abstract

Although Recurrent Neural Network (RNN) has been a powerful tool for modeling sequential data, its performance is inadequate when processing sequences with multiple patterns. In this paper, we address this challenge by introducing a novel mixture layer and constructing an adaptive RNN. The mixture layer augmented RNN (termed as M-RNN) partitions patterns in training sequences into several clusters and stores the principle patterns as prototype vectors of components in a mixture model. By leveraging the mixture layer, the proposed method can adaptively update states according to the similarities between encoded inputs and prototype vectors, leading to a stronger capacity in assimilating sequences with multiple patterns. Moreover, our approach can be further extended by taking advantage of prior knowledge about data. Experiments on both synthetic and real datasets demonstrate the effectiveness of the proposed method.

7 pages, 3 figures, 4 tables

References in corpus (9)

• Sequence to Sequence Learning with Neural Networks
• TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• Recurrent Neural Network Regularization
• Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
• Pointer Sentinel Mixture Models
• DeepStory: Video Story QA by Deep Embedded Memory Networks
• End-to-End Prediction of Buffer Overruns from Raw Source Code via Neural Memory Networks
• Chinese Poetry Generation with a Working Memory Model