Latent Alignment and Variational Attention

Abstract

Neural attention has become central to many state-of-the-art models in natural language processing and related domains. Attention networks are an easy-to-train and effective method for softly simulating alignment; however, the approach does not marginalize over latent alignments in a probabilistic sense. This property makes it difficult to compare attention to other alignment approaches, to compose it with probabilistic models, and to perform posterior inference conditioned on observed data. A related latent approach, hard attention, fixes these issues, but is generally harder to train and less accurate. This work considers variational attention networks, alternatives to soft and hard attention for learning latent variable alignment models, with tighter approximation bounds based on amortized variational inference. We further propose methods for reducing the variance of gradients to make these approaches computationally feasible. Experiments show that for machine translation and visual question answering, inefficient exact latent variable models outperform standard neural attention, but these gains go away when using hard attention based training. On the other hand, variational attention retains most of the performance gain but with training speed comparable to neural attention.

arXiv commentaccepted by NIPS 2018

Cited by in corpus (38)

• Incorporating BERT into Neural Machine Translation
• Sequential Latent Knowledge Selection for Knowledge-Grounded Dialogue
• Fixup Initialization: Residual Learning Without Normalization
• Explicit Sparse Transformer: Concentrated Attention Through Explicit Selection
• Pervasive Attention: 2D Convolutional Neural Networks for Sequence-to-Sequence Prediction
• DeLighT: Deep and Light-weight Transformer
• A Tutorial on Deep Latent Variable Models of Natural Language
• Adaptively Sparse Transformers
• Time-aware Large Kernel Convolutions
• Structured Reordering for Modeling Latent Alignments in Sequence Transduction
• Contextual Dropout: An Efficient Sample-Dependent Dropout Module
• Neural Data-to-Text Generation via Jointly Learning the Segmentation and Correspondence
• Towards Generating Long and Coherent Text with Multi-Level Latent Variable Models
• Neural Baselines for Word Alignment
• Understanding Multi-Head Attention in Abstractive Summarization
• A study of latent monotonic attention variants
• Mixture Content Selection for Diverse Sequence Generation
• Unsupervised Recurrent Neural Network Grammars
• Select and Attend: Towards Controllable Content Selection in Text Generation
• Sequence-Level Mixed Sample Data Augmentation
• An Imitation Learning Approach to Unsupervised Parsing
• Not All Attention Is Needed: Gated Attention Network for Sequence Data
• A Mixture of $h-1$ Heads is Better than $h$ Heads
• Alignment Attention by Matching Key and Query Distributions
• On the Learning Dynamics of Attention Networks
• Bayesian Attention Modules
• Cascaded Head-colliding Attention
• Context-Gated Convolution
• Learning Audio-Visual Correlations from Variational Cross-Modal Generation
• Discrete Auto-regressive Variational Attention Models for Text Modeling
• Learning Opinion Summarizers by Selecting Informative Reviews
• Generative latent neural models for automatic word alignment
• Pathologies in priors and inference for Bayesian transformers
• Autoregressive Knowledge Distillation through Imitation Learning
• Sequence-to-Sequence Learning with Latent Neural Grammars
• Multilevel Text Alignment with Cross-Document Attention
• Focus-Constrained Attention Mechanism for CVAE-based Response Generation
• Discrete Variational Attention Models for Language Generation