Dynamic Layer Aggregation for Neural Machine Translation with Routing-by-Agreement

Abstract

With the promising progress of deep neural networks, layer aggregation has been used to fuse information across layers in various fields, such as computer vision and machine translation. However, most of the previous methods combine layers in a static fashion in that their aggregation strategy is independent of specific hidden states. Inspired by recent progress on capsule networks, in this paper we propose to use routing-by-agreement strategies to aggregate layers dynamically. Specifically, the algorithm learns the probability of a part (individual layer representations) assigned to a whole (aggregated representations) in an iterative way and combines parts accordingly. We implement our algorithm on top of the state-of-the-art neural machine translation model TRANSFORMER and conduct experiments on the widely-used WMT14 English-German and WMT17 Chinese-English translation datasets. Experimental results across language pairs show that the proposed approach consistently outperforms the strong baseline model and a representative static aggregation model.

AAAI 2019

References in corpus (6)

• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Achieving Human Parity on Automatic Chinese to English News Translation
• Capsules for Object Segmentation
• Capsule Network Performance on Complex Data
• Multi-layer Representation Fusion for Neural Machine Translation
• A Deep Memory-based Architecture for Sequence-to-Sequence Learning

Cited by in corpus (5)

• milliEgo: Single-chip mmWave Radar Aided Egomotion Estimation via Deep Sensor Fusion
• Dynamic Past and Future for Neural Machine Translation
• Multi-Pass Transformer for Machine Translation
• Multi-Perspective Inferrer: Reasoning Sentences Relationship from Holistic Perspective
• Residual Tree Aggregation of Layers for Neural Machine Translation