Rich Image Captioning in the Wild

Abstract

We present an image caption system that addresses new challenges of automatically describing images in the wild. The challenges include high quality caption quality with respect to human judgments, out-of-domain data handling, and low latency required in many applications. Built on top of a state-of-the-art framework, we developed a deep vision model that detects a broad range of visual concepts, an entity recognition model that identifies celebrities and landmarks, and a confidence model for the caption output. Experimental results show that our caption engine outperforms previous state-of-the-art systems significantly on both in-domain dataset (i.e. MS COCO) and out of-domain datasets.

References in corpus (4)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Image Captioning with Semantic Attention
• Language Models for Image Captioning: The Quirks and What Works

Cited by in corpus (14)

• Multimodal Intelligence: Representation Learning, Information Fusion, and Applications
• Improved Image Captioning via Policy Gradient optimization of SPIDEr
• Boosting Image Captioning with Attributes
• Partially-Supervised Image Captioning
• Attacking Visual Language Grounding with Adversarial Examples: A Case Study on Neural Image Captioning
• Semantic Compositional Networks for Visual Captioning
• Comparison of State-of-the-Art Deep Learning APIs for Image Multi-Label Classification using Semantic Metrics
• Video Captioning with Transferred Semantic Attributes
• Entity-aware Image Caption Generation
• A Comprehensive Survey of Deep Learning for Image Captioning
• Unsupervised Image Captioning
• Vision and Language: from Visual Perception to Content Creation
• "Factual" or "Emotional": Stylized Image Captioning with Adaptive Learning and Attention
• IMG2SMI: Translating Molecular Structure Images to Simplified Molecular-input Line-entry System