Teaching Machines to Describe Images via Natural Language Feedback

Abstract

Robots will eventually be part of every household. It is thus critical to enable algorithms to learn from and be guided by non-expert users. In this paper, we bring a human in the loop, and enable a human teacher to give feedback to a learning agent in the form of natural language. We argue that a descriptive sentence can provide a much stronger learning signal than a numeric reward in that it can easily point to where the mistakes are and how to correct them. We focus on the problem of image captioning in which the quality of the output can easily be judged by non-experts. We propose a hierarchical phrase-based captioning model trained with policy gradients, and design a feedback network that provides reward to the learner by conditioning on the human-provided feedback. We show that by exploiting descriptive feedback our model learns to perform better than when given independently written human captions.

arXiv comment13 pages

References in corpus (11)

• Adam: A Method for Stochastic Optimization
• A Neural Conversational Model
• Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
• Sequence Level Training with Recurrent Neural Networks
• Improved Image Captioning via Policy Gradient optimization of SPIDEr
• Deep Reinforcement Learning for Dialogue Generation
• Towards Diverse and Natural Image Descriptions via a Conditional GAN
• Dialog-based Language Learning
• Dialogue Learning With Human-In-The-Loop
• Beating Atari with Natural Language Guided Reinforcement Learning
• The BURCHAK corpus: a Challenge Data Set for Interactive Learning of Visually Grounded Word Meanings

Cited by in corpus (12)

• Discriminability objective for training descriptive captions
• Adversarial NLI: A New Benchmark for Natural Language Understanding
• A Neural Compositional Paradigm for Image Captioning
• Situation Recognition with Graph Neural Networks
• ACTRCE: Augmenting Experience via Teacher's Advice For Multi-Goal Reinforcement Learning
• Why Build an Assistant in Minecraft?
• Detection and Description of Change in Visual Streams
• Interactive Learning from Activity Description
• Learning to Caption Images through a Lifetime by Asking Questions
• Natural Vocabulary Emerges from Free-Form Annotations
• Grounding Human-to-Vehicle Advice for Self-driving Vehicles
• Are You Sure You Want To Do That? Classification with Verification