Cooperative image captioning

Abstract

When describing images with natural language, the descriptions can be made more informative if tuned using downstream tasks. This is often achieved by training two networks: a "speaker network" that generates sentences given an image, and a "listener network" that uses them to perform a task. Unfortunately, training multiple networks jointly to communicate to achieve a joint task, faces two major challenges. First, the descriptions generated by a speaker network are discrete and stochastic, making optimization very hard and inefficient. Second, joint training usually causes the vocabulary used during communication to drift and diverge from natural language. We describe an approach that addresses both challenges. We first develop a new effective optimization based on partial-sampling from a multinomial distribution combined with straight-through gradient updates, which we name PSST for Partial-Sampling Straight-Through. Second, we show that the generated descriptions can be kept close to natural by constraining them to be similar to human descriptions. Together, this approach creates descriptions that are both more discriminative and more natural than previous approaches. Evaluations on the standard COCO benchmark show that PSST Multinomial dramatically improve the recall@10 from 60% to 86% maintaining comparable language naturalness, and human evaluations show that it also increases naturalness while keeping the discriminative power of generated captions.

References in corpus (11)

• Auto-Encoding Variational Bayes
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
• Gradient Estimation Using Stochastic Computation Graphs
• REBAR: Low-variance, unbiased gradient estimates for discrete latent variable models
• Variational inference for Monte Carlo objectives
• Backpropagation through the Void: Optimizing control variates for black-box gradient estimation
• Emergent Translation in Multi-Agent Communication
• A Neural Compositional Paradigm for Image Captioning