Bayesian Attention Modules

Abstract

Attention modules, as simple and effective tools, have not only enabled deep neural networks to achieve state-of-the-art results in many domains, but also enhanced their interpretability. Most current models use deterministic attention modules due to their simplicity and ease of optimization. Stochastic counterparts, on the other hand, are less popular despite their potential benefits. The main reason is that stochastic attention often introduces optimization issues or requires significant model changes. In this paper, we propose a scalable stochastic version of attention that is easy to implement and optimize. We construct simplex-constrained attention distributions by normalizing reparameterizable distributions, making the training process differentiable. We learn their parameters in a Bayesian framework where a data-dependent prior is introduced for regularization. We apply the proposed stochastic attention modules to various attention-based models, with applications to graph node classification, visual question answering, image captioning, machine translation, and language understanding. Our experiments show the proposed method brings consistent improvements over the corresponding baselines.

References in corpus (22)

• Adam: A Method for Stochastic Optimization
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)
• Weight Uncertainty in Neural Networks
• ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
• Sequence Level Training with Recurrent Neural Networks
• Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift
• SQuAD: 100,000+ Questions for Machine Comprehension of Text
• A Recurrent Latent Variable Model for Sequential Data
• Revisiting Semi-Supervised Learning with Graph Embeddings
• Stand-Alone Self-Attention in Vision Models
• Learning Stochastic Recurrent Networks
• Evaluating Bayesian Deep Learning Methods for Semantic Segmentation
• Latent Alignment and Variational Attention
• Discriminability objective for training descriptive captions
• WHAI: Weibull Hybrid Autoencoding Inference for Deep Topic Modeling
• Pathwise Derivatives Beyond the Reparameterization Trick
• Variational Neural Machine Translation
• Variational Attention for Sequence-to-Sequence Models
• Sequence to Sequence Mixture Model for Diverse Machine Translation
• Adaptive Correlated Monte Carlo for Contextual Categorical Sequence Generation