Fine-Grained Image Captioning with Global-Local Discriminative Objective

Abstract

Significant progress has been made in recent years in image captioning, an active topic in the fields of vision and language. However, existing methods tend to yield overly general captions and consist of some of the most frequent words/phrases, resulting in inaccurate and indistinguishable descriptions (see Figure 1). This is primarily due to (i) the conservative characteristic of traditional training objectives that drives the model to generate correct but hardly discriminative captions for similar images and (ii) the uneven word distribution of the ground-truth captions, which encourages generating highly frequent words/phrases while suppressing the less frequent but more concrete ones. In this work, we propose a novel global-local discriminative objective that is formulated on top of a reference model to facilitate generating fine-grained descriptive captions. Specifically, from a global perspective, we design a novel global discriminative constraint that pulls the generated sentence to better discern the corresponding image from all others in the entire dataset. From the local perspective, a local discriminative constraint is proposed to increase attention such that it emphasizes the less frequent but more concrete words/phrases, thus facilitating the generation of captions that better describe the visual details of the given images. We evaluate the proposed method on the widely used MS-COCO dataset, where it outperforms the baseline methods by a sizable margin and achieves competitive performance over existing leading approaches. We also conduct self-retrieval experiments to demonstrate the discriminability of the proposed method.

Accepted by TMM

References in corpus (14)

• Adam: A Method for Stochastic Optimization
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Conditional Generative Adversarial Nets
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Microsoft COCO Captions: Data Collection and Evaluation Server
• Recurrent Models of Visual Attention
• Sequence Level Training with Recurrent Neural Networks
• VSE++: Improving Visual-Semantic Embeddings with Hard Negatives
• Object-Part Attention Model for Fine-grained Image Classification
• DISC: Deep Image Saliency Computing via Progressive Representation Learning
• Actor-Critic Sequence Training for Image Captioning
• Discriminability objective for training descriptive captions
• Stack-Captioning: Coarse-to-Fine Learning for Image Captioning
• Knowledge-Embedded Representation Learning for Fine-Grained Image Recognition