Self-supervised learning of visual features through embedding images into text topic spaces

Abstract

End-to-end training from scratch of current deep architectures for new computer vision problems would require Imagenet-scale datasets, and this is not always possible. In this paper we present a method that is able to take advantage of freely available multi-modal content to train computer vision algorithms without human supervision. We put forward the idea of performing self-supervised learning of visual features by mining a large scale corpus of multi-modal (text and image) documents. We show that discriminative visual features can be learnt efficiently by training a CNN to predict the semantic context in which a particular image is more probable to appear as an illustration. For this we leverage the hidden semantic structures discovered in the text corpus with a well-known topic modeling technique. Our experiments demonstrate state of the art performance in image classification, object detection, and multi-modal retrieval compared to recent self-supervised or natural-supervised approaches.

Accepted CVPR 2017 paper

References in corpus (8)

• Caffe: Convolutional Architecture for Fast Feature Embedding
• Unsupervised Visual Representation Learning by Context Prediction
• A Comprehensive Survey on Cross-modal Retrieval
• Stacked What-Where Auto-encoders
• Unsupervised Learning of Visual Representations using Videos
• Data-dependent Initializations of Convolutional Neural Networks
• Learning to See by Moving
• Convolutional Clustering for Unsupervised Learning

Cited by in corpus (5)

• Learning to Prompt for Vision-Language Models
• Equal But Not The Same: Understanding the Implicit Relationship Between Persuasive Images and Text
• Unsupervised Single Image Deraining with Self-supervised Constraints
• TextTopicNet - Self-Supervised Learning of Visual Features Through Embedding Images on Semantic Text Spaces
• Self-Supervised Visual Representations for Cross-Modal Retrieval