Multi-branch Attentive Transformer

Abstract

While the multi-branch architecture is one of the key ingredients to the success of computer vision tasks, it has not been well investigated in natural language processing, especially sequence learning tasks. In this work, we propose a simple yet effective variant of Transformer called multi-branch attentive Transformer (briefly, MAT), where the attention layer is the average of multiple branches and each branch is an independent multi-head attention layer. We leverage two training techniques to regularize the training: drop-branch, which randomly drops individual branches during training, and proximal initialization, which uses a pre-trained Transformer model to initialize multiple branches. Experiments on machine translation, code generation and natural language understanding demonstrate that such a simple variant of Transformer brings significant improvements. Our code is available at \url{https://github.com/HA-Transformer}.

17 pages

References in corpus (9)

• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• DARTS: Differentiable Architecture Search
• Efficient Neural Architecture Search via Parameter Sharing
• Pay Less Attention with Lightweight and Dynamic Convolutions
• The Evolved Transformer
• Weighted Transformer Network for Machine Translation
• RACE: Large-scale ReAding Comprehension Dataset From Examinations
• Code Generation as a Dual Task of Code Summarization
• MUSE: Parallel Multi-Scale Attention for Sequence to Sequence Learning

Cited by in corpus (4)

• A Simple but Tough-to-Beat Data Augmentation Approach for Natural Language Understanding and Generation
• Diverse Pretrained Context Encodings Improve Document Translation
• BERT, mBERT, or BiBERT? A Study on Contextualized Embeddings for Neural Machine Translation
• BERT-JAM: Boosting BERT-Enhanced Neural Machine Translation with Joint Attention