Goal-driven text descriptions for images

Abstract

A big part of achieving Artificial General Intelligence(AGI) is to build a machine that can see and listen like humans. Much work has focused on designing models for image classification, video classification, object detection, pose estimation, speech recognition, etc., and has achieved significant progress in recent years thanks to deep learning. However, understanding the world is not enough. An AI agent also needs to know how to talk, especially how to communicate with a human. While perception (vision, for example) is more common across animal species, the use of complicated language is unique to humans and is one of the most important aspects of intelligence. In this thesis, we focus on generating textual output given visual input. In Chapter 3, we focus on generating the referring expression, a text description for an object in the image so that a receiver can infer which object is being described. We use a comprehension machine to directly guide the generated referring expressions to be more discriminative. In Chapter 4, we introduce a method that encourages discriminability in image caption generation. We show that more discriminative captioning models generate more descriptive captions. In Chapter 5, we study how training objectives and sampling methods affect the models' ability to generate diverse captions. We find that a popular captioning training strategy will be detrimental to the diversity of generated captions. In Chapter 6, we propose a model that can control the length of generated captions. By changing the desired length, one can influence the style and descriptiveness of the captions. Finally, in Chapter 7, we rank/generate informative image tags according to their information utility. The proposed method better matches what humans think are the most important tags for the images.

arXiv commentPh.D. thesis

References in corpus (27)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Teaching Machines to Read and Comprehend
• Microsoft COCO Captions: Data Collection and Evaluation Server
• Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
• Evaluating Large Language Models Trained on Code
• Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
• The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
• CNN: Single-label to Multi-label
• CTRL: A Conditional Transformer Language Model for Controllable Generation
• Improved Image Captioning via Policy Gradient optimization of SPIDEr
• Towards Automatic Learning of Procedures from Web Instructional Videos
• Context-Aware Visual Policy Network for Sequence-Level Image Captioning
• Actor-Critic Sequence Training for Image Captioning
• Mutual Information and Diverse Decoding Improve Neural Machine Translation
• Mixture Models for Diverse Machine Translation: Tricks of the Trade
• CLEVR-Dialog: A Diagnostic Dataset for Multi-Round Reasoning in Visual Dialog
• Image Captioning at Will: A Versatile Scheme for Effectively Injecting Sentiments into Image Descriptions
• A Better Variant of Self-Critical Sequence Training
• Training and Evaluating Multimodal Word Embeddings with Large-scale Web Annotated Images
• Elucidating image-to-set prediction: An analysis of models, losses and datasets
• Summarize, Outline, and Elaborate: Long-Text Generation via Hierarchical Supervision from Extractive Summaries
• CapWAP: Captioning with a Purpose
• Detection and Description of Change in Visual Streams
• SMURF: SeMantic and linguistic UndeRstanding Fusion for Caption Evaluation via Typicality Analysis