Weighted Transformer Network for Machine Translation

Abstract

State-of-the-art results on neural machine translation often use attentional sequence-to-sequence models with some form of convolution or recursion. Vaswani et al. (2017) propose a new architecture that avoids recurrence and convolution completely. Instead, it uses only self-attention and feed-forward layers. While the proposed architecture achieves state-of-the-art results on several machine translation tasks, it requires a large number of parameters and training iterations to converge. We propose Weighted Transformer, a Transformer with modified attention layers, that not only outperforms the baseline network in BLEU score but also converges 15-40% faster. Specifically, we replace the multi-head attention by multiple self-attention branches that the model learns to combine during the training process. Our model improves the state-of-the-art performance by 0.5 BLEU points on the WMT 2014 English-to-German translation task and by 0.4 on the English-to-French translation task.

References in corpus (14)

• Adam: A Method for Stochastic Optimization
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• Convolutional Sequence to Sequence Learning
• A Structured Self-attentive Sentence Embedding
• A Deep Reinforced Model for Abstractive Summarization
• Regularizing and Optimizing LSTM Language Models
• Aggregated Residual Transformations for Deep Neural Networks
• Quasi-Recurrent Neural Networks
• Neural Machine Translation in Linear Time
• Shake-Shake regularization
• Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modeling
• A Decomposable Attention Model for Natural Language Inference
• A Convolutional Encoder Model for Neural Machine Translation

Cited by in corpus (35)

• Natural Language Processing Advancements By Deep Learning: A Survey
• fairseq: A Fast, Extensible Toolkit for Sequence Modeling
• Understanding and Improving Transformer From a Multi-Particle Dynamic System Point of View
• Understanding Back-Translation at Scale
• A Simple but Tough-to-Beat Data Augmentation Approach for Natural Language Understanding and Generation
• Are Sixteen Heads Really Better than One?
• Data Diversification: A Simple Strategy For Neural Machine Translation
• Multi-Head Attention with Disagreement Regularization
• Joint Source-Target Self Attention with Locality Constraints
• Margin-based Parallel Corpus Mining with Multilingual Sentence Embeddings
• Multi-branch Attentive Transformer
• Information Aggregation for Multi-Head Attention with Routing-by-Agreement
• Hybrid Self-Attention Network for Machine Translation
• Scalable Transformers for Neural Machine Translation
• Residual Attention Net for Superior Cross-Domain Time Sequence Modeling
• Mask Attention Networks: Rethinking and Strengthen Transformer
• Recurrent Graph Syntax Encoder for Neural Machine Translation
• BiQGEMM: Matrix Multiplication with Lookup Table For Binary-Coding-based Quantized DNNs
• Scheduled DropHead: A Regularization Method for Transformer Models
• Multi-Unit Transformers for Neural Machine Translation
• Neural Language Priors
• Multiresolution Transformer Networks: Recurrence is Not Essential for Modeling Hierarchical Structure
• Improving Self-supervised Pre-training via a Fully-Explored Masked Language Model
• Fast and Simple Mixture of Softmaxes with BPE and Hybrid-LightRNN for Language Generation
• IOT: Instance-wise Layer Reordering for Transformer Structures
• Query-Key Normalization for Transformers
• Meta-Embeddings Based On Self-Attention
• Neural Machine Translation: A Review and Survey
• asya: Mindful verbal communication using deep learning
• Automatic Acrostic Couplet Generation with Three-Stage Neural Network Pipelines
• UniDrop: A Simple yet Effective Technique to Improve Transformer without Extra Cost
• Neuron Interaction Based Representation Composition for Neural Machine Translation
• Generative Models for Security: Attacks, Defenses, and Opportunities
• The University of Sydney's Machine Translation System for WMT19
• Improving Multi-Head Attention with Capsule Networks