Hard but Robust, Easy but Sensitive: How Encoder and Decoder Perform in Neural Machine Translation
arXiv:1908.06259
Abstract
Neural machine translation (NMT) typically adopts the encoder-decoder framework. A good understanding of the characteristics and functionalities of the encoder and decoder can help to explain the pros and cons of the framework, and design better models for NMT. In this work, we conduct an empirical study on the encoder and the decoder in NMT, taking Transformer as an example. We find that 1) the decoder handles an easier task than the encoder in NMT, 2) the decoder is more sensitive to the input noise than the encoder, and 3) the preceding words/tokens in the decoder provide strong conditional information, which accounts for the two observations above. We hope those observations can shed light on the characteristics of the encoder and decoder and inspire future research on NMT.
References in corpus (15)
- Sequence to Sequence Learning with Neural Networks
- Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
- Convolutional Sequence to Sequence Learning
- Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
- Sequence Level Training with Recurrent Neural Networks
- Effective Approaches to Attention-based Neural Machine Translation
- MASS: Masked Sequence to Sequence Pre-training for Language Generation
- Achieving Human Parity on Automatic Chinese to English News Translation
- Non-Autoregressive Neural Machine Translation
- Universal Transformers
- Sharp Nearby, Fuzzy Far Away: How Neural Language Models Use Context
- Non-Autoregressive Neural Machine Translation with Enhanced Decoder Input
- Hybrid Self-Attention Network for Machine Translation
- Beyond Error Propagation in Neural Machine Translation: Characteristics of Language Also Matter
- Dense Information Flow for Neural Machine Translation