Learning Visual Question Answering by Bootstrapping Hard Attention
arXiv:1808.00300
Abstract
Attention mechanisms in biological perception are thought to select subsets of perceptual information for more sophisticated processing which would be prohibitive to perform on all sensory inputs. In computer vision, however, there has been relatively little exploration of hard attention, where some information is selectively ignored, in spite of the success of soft attention, where information is re-weighted and aggregated, but never filtered out. Here, we introduce a new approach for hard attention and find it achieves very competitive performance on a recently-released visual question answering datasets, equalling and in some cases surpassing similar soft attention architectures while entirely ignoring some features. Even though the hard attention mechanism is thought to be non-differentiable, we found that the feature magnitudes correlate with semantic relevance, and provide a useful signal for our mechanism's attentional selection criterion. Because hard attention selects important features of the input information, it can also be more efficient than analogous soft attention mechanisms. This is especially important for recent approaches that use non-local pairwise operations, whereby computational and memory costs are quadratic in the size of the set of features.
ECCV 2018
References in corpus (25)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
- Deep Residual Learning for Image Recognition
- Categorical Reparameterization with Gumbel-Softmax
- Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
- VQA: Visual Question Answering
- The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
- Dynamic Memory Networks for Visual and Textual Question Answering
- Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
- Modulating early visual processing by language
- ABC-CNN: An Attention Based Convolutional Neural Network for Visual Question Answering
- Are You Talking to a Machine? Dataset and Methods for Multilingual Image Question Answering
- Transparency by Design: Closing the Gap Between Performance and Interpretability in Visual Reasoning
- Ask Your Neurons: A Neural-based Approach to Answering Questions about Images
- Non-local Neural Networks
- Show, Ask, Attend, and Answer: A Strong Baseline For Visual Question Answering
- A Focused Dynamic Attention Model for Visual Question Answering
- Compositional Attention Networks for Machine Reasoning
- Learning to Reason: End-to-End Module Networks for Visual Question Answering
- Hyperbolic Attention Networks
- Dynamic Neural Turing Machine with Soft and Hard Addressing Schemes
- Structured Attentions for Visual Question Answering
- DDRprog: A CLEVR Differentiable Dynamic Reasoning Programmer
- Hard to Cheat: A Turing Test based on Answering Questions about Images
- Mean Box Pooling: A Rich Image Representation and Output Embedding for the Visual Madlibs Task