Saliency-based Sequential Image Attention with Multiset Prediction
arXiv:1711.05165
Abstract
Humans process visual scenes selectively and sequentially using attention. Central to models of human visual attention is the saliency map. We propose a hierarchical visual architecture that operates on a saliency map and uses a novel attention mechanism to sequentially focus on salient regions and take additional glimpses within those regions. The architecture is motivated by human visual attention, and is used for multi-label image classification on a novel multiset task, demonstrating that it achieves high precision and recall while localizing objects with its attention. Unlike conventional multi-label image classification models, the model supports multiset prediction due to a reinforcement-learning based training process that allows for arbitrary label permutation and multiple instances per label.
To appear in Advances in Neural Information Processing Systems 30 (NIPS 2017)
References in corpus (9)
- Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation
- Multiple Object Recognition with Visual Attention
- Hierarchical Object Detection with Deep Reinforcement Learning
- Generating Visual Explanations
- Spatial Transformer Networks
- Emergence of foveal image sampling from learning to attend in visual scenes
- Localizing by Describing: Attribute-Guided Attention Localization for Fine-Grained Recognition
- Paying More Attention to Saliency: Image Captioning with Saliency and Context Attention
- Towards Instance Segmentation with Object Priority: Prominent Object Detection and Recognition