Improving Conditioning in Context-Aware Sequence to Sequence Models

Abstract

Neural sequence to sequence models are well established for applications which can be cast as mapping a single input sequence into a single output sequence. In this work, we focus on cases where generation is conditioned on both a short query and a long context, such as abstractive question answering or document-level translation. We modify the standard sequence-to-sequence approach to make better use of both the query and the context by expanding the conditioning mechanism to intertwine query and context attention. We also introduce a simple and efficient data augmentation method for the proposed model. Experiments on three different tasks show that both changes lead to consistent improvements.

References in corpus (5)

• End-To-End Memory Networks
• fairseq: A Fast, Extensible Toolkit for Sequence Modeling
• Context in Neural Machine Translation: A Review of Models and Evaluations
• Improving the Transformer Translation Model with Document-Level Context
• ELI5: Long Form Question Answering

Cited by in corpus (4)

• Unstructured Text Enhanced Open-domain Dialogue System: A Systematic Survey
• A Survey on Document-level Neural Machine Translation: Methods and Evaluation
• SJTU-NICT's Supervised and Unsupervised Neural Machine Translation Systems for the WMT20 News Translation Task
• Policy-driven Knowledge Selection and Response Generation for Document-grounded Dialogue