Sequence Generation: From Both Sides to the Middle
arXiv:1906.09601
Abstract
The encoder-decoder framework has achieved promising process for many sequence generation tasks, such as neural machine translation and text summarization. Such a framework usually generates a sequence token by token from left to right, hence (1) this autoregressive decoding procedure is time-consuming when the output sentence becomes longer, and (2) it lacks the guidance of future context which is crucial to avoid under translation. To alleviate these issues, we propose a synchronous bidirectional sequence generation (SBSG) model which predicts its outputs from both sides to the middle simultaneously. In the SBSG model, we enable the left-to-right (L2R) and right-to-left (R2L) generation to help and interact with each other by leveraging interactive bidirectional attention network. Experiments on neural machine translation (En-De, Ch-En, and En-Ro) and text summarization tasks show that the proposed model significantly speeds up decoding while improving the generation quality compared to the autoregressive Transformer.
Accepted by IJCAI 2019
References in corpus (7)
- Sequence to Sequence Learning with Neural Networks
- Convolutional Sequence to Sequence Learning
- Non-Autoregressive Neural Machine Translation
- Selective Encoding for Abstractive Sentence Summarization
- Fast Decoding in Sequence Models using Discrete Latent Variables
- Twin Networks: Matching the Future for Sequence Generation
- Semi-Autoregressive Neural Machine Translation