The Neural Noisy Channel

Abstract

We formulate sequence to sequence transduction as a noisy channel decoding problem and use recurrent neural networks to parameterise the source and channel models. Unlike direct models which can suffer from explaining-away effects during training, noisy channel models must produce outputs that explain their inputs, and their component models can be trained with not only paired training samples but also unpaired samples from the marginal output distribution. Using a latent variable to control how much of the conditioning sequence the channel model needs to read in order to generate a subsequent symbol, we obtain a tractable and effective beam search decoder. Experimental results on abstractive sentence summarisation, morphological inflection, and machine translation show that noisy channel models outperform direct models, and that they significantly benefit from increased amounts of unpaired output data that direct models cannot easily use.

ICLR 2017

References in corpus (4)

• Sequence Transduction with Recurrent Neural Networks
• Neural Machine Translation with Reconstruction
• Language as a Latent Variable: Discrete Generative Models for Sentence Compression
• Learning to Translate in Real-time with Neural Machine Translation

Cited by in corpus (10)

• The Curious Case of Neural Text Degeneration
• Plug and Play Language Models: A Simple Approach to Controlled Text Generation
• Domain Robustness in Neural Machine Translation
• A Tutorial on Deep Latent Variable Models of Natural Language
• Encoder-Agnostic Adaptation for Conditional Language Generation
• Unified Pragmatic Models for Generating and Following Instructions
• A Comparison of Approaches to Document-level Machine Translation
• A study of latent monotonic attention variants
• Simple Unsupervised Summarization by Contextual Matching
• Neural Machine Translation: A Review and Survey