Exploring Nearest Neighbor Approaches for Image Captioning

Abstract

We explore a variety of nearest neighbor baseline approaches for image captioning. These approaches find a set of nearest neighbor images in the training set from which a caption may be borrowed for the query image. We select a caption for the query image by finding the caption that best represents the "consensus" of the set of candidate captions gathered from the nearest neighbor images. When measured by automatic evaluation metrics on the MS COCO caption evaluation server, these approaches perform as well as many recent approaches that generate novel captions. However, human studies show that a method that generates novel captions is still preferred over the nearest neighbor approach.

References in corpus (11)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Caffe: Convolutional Architecture for Fast Feature Embedding
• Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
• Deep Fragment Embeddings for Bidirectional Image Sentence Mapping
• Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN)
• Explain Images with Multimodal Recurrent Neural Networks
• Language Models for Image Captioning: The Quirks and What Works
• From Captions to Visual Concepts and Back
• CIDEr: Consensus-based Image Description Evaluation
• Simple Image Description Generator via a Linear Phrase-Based Approach

Cited by in corpus (43)

• Show and Tell: Lessons learned from the 2015 MSCOCO Image Captioning Challenge
• Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN)
• Exploring Models and Data for Image Question Answering
• Simple Baseline for Visual Question Answering
• Visual Question Answering: Datasets, Algorithms, and Future Challenges
• Image Captioning with Semantic Attention
• Cognitive Science in the era of Artificial Intelligence: A roadmap for reverse-engineering the infant language-learner
• Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding
• The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes
• Clinically Accurate Chest X-Ray Report Generation
• Generation and Comprehension of Unambiguous Object Descriptions
• Towards Diverse and Natural Image Descriptions via a Conditional GAN
• Diverse and Accurate Image Description Using a Variational Auto-Encoder with an Additive Gaussian Encoding Space
• Adversarial NLI: A New Benchmark for Natural Language Understanding
• Learning like a Child: Fast Novel Visual Concept Learning from Sentence Descriptions of Images
• Knowing When to Look: Adaptive Attention via A Visual Sentinel for Image Captioning
• On the effectiveness of task granularity for transfer learning
• Detecting Hate Speech in Multi-modal Memes
• Yin and Yang: Balancing and Answering Binary Visual Questions
• A Multimodal Memes Classification: A Survey and Open Research Issues
• 3G structure for image caption generation
• Search Engine Guided Non-Parametric Neural Machine Translation
• A Neural Compositional Paradigm for Image Captioning
• Blindfold Baselines for Embodied QA
• Context-aware Captions from Context-agnostic Supervision
• In Defense of Grid Features for Visual Question Answering
• Skeleton Key: Image Captioning by Skeleton-Attribute Decomposition
• ChartNet: Visual Reasoning over Statistical Charts using MAC-Networks
• Comprehensive Image Captioning via Scene Graph Decomposition
• Fast, Diverse and Accurate Image Captioning Guided By Part-of-Speech
• Going Beneath the Surface: Evaluating Image Captioning for Grammaticality, Truthfulness and Diversity
• Visual Referring Expression Recognition: What Do Systems Actually Learn?
• Text-guided Attention Model for Image Captioning
• Commonly Uncommon: Semantic Sparsity in Situation Recognition
• Tinkering Under the Hood: Interactive Zero-Shot Learning with Net Surgery
• Challenges and Prospects in Vision and Language Research
• Reasoning about Fine-grained Attribute Phrases using Reference Games
• Improving Image Captioning by Leveraging Knowledge Graphs
• The Wisdom of MaSSeS: Majority, Subjectivity, and Semantic Similarity in the Evaluation of VQA
• SPICE: Semantic Propositional Image Caption Evaluation
• Neural Abstructions: Abstractions that Support Construction for Grounded Language Learning
• Automated Image Captioning for Rapid Prototyping and Resource Constrained Environments
• Cross-Modal Retrieval Augmentation for Multi-Modal Classification