Talking-Heads Attention

Abstract

We introduce "talking-heads attention" - a variation on multi-head attention which includes linearprojections across the attention-heads dimension, immediately before and after the softmax operation.While inserting only a small number of additional parameters and a moderate amount of additionalcomputation, talking-heads attention leads to better perplexities on masked language modeling tasks, aswell as better quality when transfer-learning to language comprehension and question answering tasks.

References in corpus (2)

• Neural Machine Translation by Jointly Learning to Align and Translate
• Adafactor: Adaptive Learning Rates with Sublinear Memory Cost

Cited by in corpus (8)

• Global Attention based Graph Convolutional Neural Networks for Improved Materials Property Prediction
• Depth-Wise Convolutions in Vision Transformers for Efficient Training on Small Datasets
• Vision Transformers with Patch Diversification
• Refiner: Refining Self-attention for Vision Transformers
• Dual-interest Factorization-heads Attention for Sequential Recommendation
• Predicting Rigid Body Dynamics using Dual Quaternion Recurrent Neural Networks with Quaternion Attention
• MedGPT: Medical Concept Prediction from Clinical Narratives
• WeChat Neural Machine Translation Systems for WMT21