Tell-the-difference: Fine-grained Visual Descriptor via a Discriminating Referee
arXiv:1910.06426
Abstract
In this paper, we investigate a novel problem of telling the difference between image pairs in natural language. Compared to previous approaches for single image captioning, it is challenging to fetch linguistic representation from two independent visual information. To this end, we have proposed an effective encoder-decoder caption framework based on Hyper Convolution Net. In addition, a series of novel feature fusing techniques for pairwise visual information fusing are introduced and a discriminating referee is proposed to evaluate the pipeline. Because of the lack of appropriate datasets to support this task, we have collected and annotated a large new dataset with Amazon Mechanical Turk (AMT) for generating captions in a pairwise manner (with 14764 images and 26710 image pairs in total). The dataset is the first one on the relative difference caption task that provides descriptions in free language. We evaluate the effectiveness of our model on two datasets in the field and it outperforms the state-of-the-art approach by a large margin.
References in corpus (13)
- Adam: A Method for Stochastic Optimization
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- Sequence to Sequence Learning with Neural Networks
- Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
- Texture Networks: Feed-forward Synthesis of Textures and Stylized Images
- Improved Image Captioning via Policy Gradient optimization of SPIDEr
- Semantic Jitter: Dense Supervision for Visual Comparisons via Synthetic Images
- Detecting Visual Relationships with Deep Relational Networks
- Deep Reinforcement Learning-based Image Captioning with Embedding Reward
- Reasoning About Pragmatics with Neural Listeners and Speakers
- Dynamic Computational Time for Visual Attention
- Reasoning about Fine-grained Attribute Phrases using Reference Games