Reasoning about Fine-grained Attribute Phrases using Reference Games
arXiv:1708.08874
Abstract
We present a framework for learning to describe fine-grained visual differences between instances using attribute phrases. Attribute phrases capture distinguishing aspects of an object (e.g., "propeller on the nose" or "door near the wing" for airplanes) in a compositional manner. Instances within a category can be described by a set of these phrases and collectively they span the space of semantic attributes for a category. We collect a large dataset of such phrases by asking annotators to describe several visual differences between a pair of instances within a category. We then learn to describe and ground these phrases to images in the context of a *reference game* between a speaker and a listener. The goal of a speaker is to describe attributes of an image that allows the listener to correctly identify it within a pair. Data collected in a pairwise manner improves the ability of the speaker to generate, and the ability of the listener to interpret visual descriptions. Moreover, due to the compositionality of attribute phrases, the trained listeners can interpret descriptions not seen during training for image retrieval, and the speakers can generate attribute-based explanations for differences between previously unseen categories. We also show that embedding an image into the semantic space of attribute phrases derived from listeners offers 20% improvement in accuracy over existing attribute-based representations on the FGVC-aircraft dataset.
To appear in ICCV 2017
References in corpus (16)
- Adam: A Method for Stochastic Optimization
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
- Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
- Fine-Grained Visual Classification of Aircraft
- VQA: Visual Question Answering
- Show and Tell: A Neural Image Caption Generator
- Exploring Nearest Neighbor Approaches for Image Captioning
- Generation and Comprehension of Unambiguous Object Descriptions
- Modeling Context in Referring Expressions
- CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning
- Reasoning About Pragmatics with Neural Listeners and Speakers
- Yin and Yang: Balancing and Answering Binary Visual Questions
- Natural Language Object Retrieval
- Context-aware Captions from Context-agnostic Supervision
- Comprehension-guided referring expressions