Weighted Transformer Network for Machine Translation

Abstract

State-of-the-art results on neural machine translation often use attentional sequence-to-sequence models with some form of convolution or recursion. Vaswani et al. (2017) propose a new architecture that avoids recurrence and convolution completely. Instead, it uses only self-attention and feed-forward layers. While the proposed architecture achieves state-of-the-art results on several machine translation tasks, it requires a large number of parameters and training iterations to converge. We propose Weighted Transformer, a Transformer with modified attention layers, that not only outperforms the baseline network in BLEU score but also converges 15-40% faster. Specifically, we replace the multi-head attention by multiple self-attention branches that the model learns to combine during the training process. Our model improves the state-of-the-art performance by 0.5 BLEU points on the WMT 2014 English-to-German translation task and by 0.4 on the English-to-French translation task.

References in corpus (8)

• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• A Structured Self-attentive Sentence Embedding
• Regularizing and Optimizing LSTM Language Models
• Quasi-Recurrent Neural Networks
• Neural Machine Translation in Linear Time
• Shake-Shake regularization
• Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modeling

Cited by in corpus (6)

• fairseq: A Fast, Extensible Toolkit for Sequence Modeling
• Understanding and Improving Transformer From a Multi-Particle Dynamic System Point of View
• Joint Source-Target Self Attention with Locality Constraints
• Information Aggregation for Multi-Head Attention with Routing-by-Agreement
• Automatic Acrostic Couplet Generation with Three-Stage Neural Network Pipelines
• The University of Sydney's Machine Translation System for WMT19