Normalized Attention Without Probability Cage
arXiv:2005.09561
Abstract
Attention architectures are widely used; they recently gained renewed popularity with Transformers yielding a streak of state of the art results. Yet, the geometrical implications of softmax-attention remain largely unexplored. In this work we highlight the limitations of constraining attention weights to the probability simplex and the resulting convex hull of value vectors. We show that Transformers are sequence length dependent biased towards token isolation at initialization and contrast Transformers to simple max- and sum-pooling - two strong baselines rarely reported. We propose to replace the softmax in self-attention with normalization, yielding a hyperparameter and data-bias robust, generally applicable architecture. We support our insights with empirical results from more than 25,000 trained models. All results and implementations are made available.
Preprint, work in progress. Feedback welcome
References in corpus (17)
- Adam: A Method for Stochastic Optimization
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
- Gaussian Error Linear Units (GELUs)
- Instance Normalization: The Missing Ingredient for Fast Stylization
- Self-Attention Generative Adversarial Networks
- Attention is not Explanation
- Universal Transformers
- Pay Less Attention with Lightweight and Dynamic Convolutions
- ReZero is All You Need: Fast Convergence at Large Depth
- Stabilizing Transformers for Reinforcement Learning
- Lite Transformer with Long-Short Range Attention
- What Can Neural Networks Reason About?
- Are Transformers universal approximators of sequence-to-sequence functions?
- Budgeted Training: Rethinking Deep Neural Network Training Under Resource Constraints
- Understanding the Difficulty of Training Transformers
- Coherent Gradients: An Approach to Understanding Generalization in Gradient Descent-based Optimization