Exploiting Deep Representations for Neural Machine Translation
arXiv:1810.10181
Abstract
Advanced neural machine translation (NMT) models generally implement encoder and decoder as multiple layers, which allows systems to model complex functions and capture complicated linguistic structures. However, only the top layers of encoder and decoder are leveraged in the subsequent process, which misses the opportunity to exploit the useful information embedded in other layers. In this work, we propose to simultaneously expose all of these signals with layer aggregation and multi-layer attention mechanisms. In addition, we introduce an auxiliary regularization term to encourage different layers to capture diverse information. Experimental results on widely-used WMT14 English-German and WMT17 Chinese-English translation data demonstrate the effectiveness and universality of the proposed approach.
arXiv commentEMNLP 2018
References in corpus (10)
- Sequence to Sequence Learning with Neural Networks
- Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
- Convolutional Sequence to Sequence Learning
- Effective Approaches to Attention-based Neural Machine Translation
- Achieving Human Parity on Automatic Chinese to English News Translation
- Modeling Coverage for Neural Machine Translation
- Convergent Learning: Do different neural networks learn the same representations?
- THUMT: An Open Source Toolkit for Neural Machine Translation
- A Deep Memory-based Architecture for Sequence-to-Sequence Learning
- Multi-Head Attention with Disagreement Regularization
Cited by in corpus (10)
- PingAn-VCGroup's Solution for ICDAR 2021 Competition on Scientific Literature Parsing Task B: Table Recognition to HTML
- Context-Aware Self-Attention Networks
- Multi-Head Attention with Disagreement Regularization
- Convolutional Self-Attention Networks
- Information Aggregation for Multi-Head Attention with Routing-by-Agreement
- Acquiring Knowledge from Pre-trained Model to Neural Machine Translation
- Improving Neural Machine Translation with Pre-trained Representation
- Graph-to-Sequence Neural Machine Translation
- Residual Tree Aggregation of Layers for Neural Machine Translation
- GRET: Global Representation Enhanced Transformer