GRET: Global Representation Enhanced Transformer

Abstract

Transformer, based on the encoder-decoder framework, has achieved state-of-the-art performance on several natural language generation tasks. The encoder maps the words in the input sentence into a sequence of hidden states, which are then fed into the decoder to generate the output sentence. These hidden states usually correspond to the input words and focus on capturing local information. However, the global (sentence level) information is seldom explored, leaving room for the improvement of generation quality. In this paper, we propose a novel global representation enhanced Transformer (GRET) to explicitly model global representation in the Transformer network. Specifically, in the proposed model, an external state is generated for the global representation from the encoder. The global representation is then fused into the decoder during the decoding process to improve generation quality. We conduct experiments in two text generation tasks: machine translation and text summarization. Experimental results on four WMT machine translation tasks and LCSTS text summarization task demonstrate the effectiveness of the proposed approach on natural language generation.

Accepted by AAAI 2020

References in corpus (19)

• Adam: A Method for Stochastic Optimization
• Neural Machine Translation by Jointly Learning to Align and Translate
• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• Convolutional Sequence to Sequence Learning
• Effective Approaches to Attention-based Neural Machine Translation
• Achieving Human Parity on Automatic Chinese to English News Translation
• Non-Autoregressive Neural Machine Translation
• Incorporating Copying Mechanism in Sequence-to-Sequence Learning
• Investigating Capsule Networks with Dynamic Routing for Text Classification
• A Convolutional Encoder Model for Neural Machine Translation
• Actor-Critic based Training Framework for Abstractive Summarization
• Multi-layer Representation Fusion for Neural Machine Translation
• Sentence-State LSTM for Text Representation
• Neural Headline Generation with Sentence-wise Optimization
• Exploiting Deep Representations for Neural Machine Translation
• Global Encoding for Abstractive Summarization
• Deconvolution-Based Global Decoding for Neural Machine Translation
• Neural Machine Translation with Word Predictions
• Dynamic Past and Future for Neural Machine Translation