Love Thy Neighbors: Image Annotation by Exploiting Image Metadata

Abstract

Some images that are difficult to recognize on their own may become more clear in the context of a neighborhood of related images with similar social-network metadata. We build on this intuition to improve multilabel image annotation. Our model uses image metadata nonparametrically to generate neighborhoods of related images using Jaccard similarities, then uses a deep neural network to blend visual information from the image and its neighbors. Prior work typically models image metadata parametrically, in contrast, our nonparametric treatment allows our model to perform well even when the vocabulary of metadata changes between training and testing. We perform comprehensive experiments on the NUS-WIDE dataset, where we show that our model outperforms state-of-the-art methods for multilabel image annotation even when our model is forced to generalize to new types of metadata.

Accepted to ICCV 2015

References in corpus (5)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Improving neural networks by preventing co-adaptation of feature detectors
• Caffe: Convolutional Architecture for Fast Feature Embedding
• Webly Supervised Learning of Convolutional Networks
• Collaborative Feature Learning from Social Media

Cited by in corpus (14)

• Automatic Image Annotation via Label Transfer in the Semantic Space
• Visual Affordance and Function Understanding: A Survey
• Geometry in Active Learning for Binary and Multi-class Image Segmentation
• Multimodal Multitask Representation Learning for Pathology Biobank Metadata Prediction
• Graph Convolutional Networks for Classification with a Structured Label Space
• Context Aware Image Annotation in Active Learning
• Structured Label Inference for Visual Understanding
• VITAL: A Visual Interpretation on Text with Adversarial Learning for Image Labeling
• Beyond Pixels: Image Provenance Analysis Leveraging Metadata
• Measuring and Predicting Tag Importance for Image Retrieval
• Deep Neural Networks In Fully Connected CRF For Image Labeling With Social Network Metadata
• Image Annotation using Multi-Layer Sparse Coding
• Zero-Shot Learning with Sparse Attribute Propagation
• Learning Joint Representations of Videos and Sentences with Web Image Search