A Deep Memory-based Architecture for Sequence-to-Sequence Learning

Abstract

We propose DEEPMEMORY, a novel deep architecture for sequence-to-sequence learning, which performs the task through a series of nonlinear transformations from the representation of the input sequence (e.g., a Chinese sentence) to the final output sequence (e.g., translation to English). Inspired by the recently proposed Neural Turing Machine (Graves et al., 2014), we store the intermediate representations in stacked layers of memories, and use read-write operations on the memories to realize the nonlinear transformations between the representations. The types of transformations are designed in advance but the parameters are learned from data. Through layer-by-layer transformations, DEEPMEMORY can model complicated relations between sequences necessary for applications such as machine translation between distant languages. The architecture can be trained with normal back-propagation on sequenceto-sequence data, and the learning can be easily scaled up to a large corpus. DEEPMEMORY is broad enough to subsume the state-of-the-art neural translation model in (Bahdanau et al., 2015) as its special case, while significantly improving upon the model with its deeper architecture. Remarkably, DEEPMEMORY, being purely neural network-based, can achieve performance comparable to the traditional phrase-based machine translation system Moses with a small vocabulary and a modest parameter size.

13 pages, Under review as a conference paper at ICLR 2016

References in corpus (11)

• Sequence to Sequence Learning with Neural Networks
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• ADADELTA: An Adaptive Learning Rate Method
• Natural Language Processing (almost) from Scratch
• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• DRAW: A Recurrent Neural Network For Image Generation
• Effective Approaches to Attention-based Neural Machine Translation
• Grammar as a Foreign Language
• Towards Neural Network-based Reasoning
• Neural Enquirer: Learning to Query Tables with Natural Language
• $gen$CNN: A Convolutional Architecture for Word Sequence Prediction

Cited by in corpus (8)

• Long Short-Term Memory-Networks for Machine Reading
• Emotional Chatting Machine: Emotional Conversation Generation with Internal and External Memory
• Memory-enhanced Decoder for Neural Machine Translation
• Exploiting Deep Representations for Neural Machine Translation
• Neural Language Generation: Formulation, Methods, and Evaluation
• Dynamic Layer Aggregation for Neural Machine Translation with Routing-by-Agreement
• Neural Machine Translation Advised by Statistical Machine Translation
• Multiscale Collaborative Deep Models for Neural Machine Translation