Towards Diverse and Accurate Image Captions via Reinforcing Determinantal Point Process

Abstract

Although significant progress has been made in the field of automatic image captioning, it is still a challenging task. Previous works normally pay much attention to improving the quality of the generated captions but ignore the diversity of captions. In this paper, we combine determinantal point process (DPP) and reinforcement learning (RL) and propose a novel reinforcing DPP (R-DPP) approach to generate a set of captions with high quality and diversity for an image. We show that R-DPP performs better on accuracy and diversity than using noise as a control signal (GANs, VAEs). Moreover, R-DPP is able to preserve the modes of the learned distribution. Hence, beam search algorithm can be applied to generate a single accurate caption, which performs better than other RL-based models.

arXiv comment14 pages. Code is comming soon,please pay attention to my personal page visal.cs.cityu.edu.hk/people/qingzhong-wang/

References in corpus (7)

• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering
• Fast Greedy MAP Inference for Determinantal Point Process to Improve Recommendation Diversity
• Discriminability objective for training descriptive captions
• Describing like humans: on diversity in image captioning
• Fast, Diverse and Accurate Image Captioning Guided By Part-of-Speech
• Show, Control and Tell: A Framework for Generating Controllable and Grounded Captions

Cited by in corpus (3)

• Diverse Image Captioning with Context-Object Split Latent Spaces
• Compare and Reweight: Distinctive Image Captioning Using Similar Images Sets
• Group-based Distinctive Image Captioning with Memory Attention