Transitive Invariance for Self-supervised Visual Representation Learning

Abstract

Learning visual representations with self-supervised learning has become popular in computer vision. The idea is to design auxiliary tasks where labels are free to obtain. Most of these tasks end up providing data to learn specific kinds of invariance useful for recognition. In this paper, we propose to exploit different self-supervised approaches to learn representations invariant to (i) inter-instance variations (two objects in the same class should have similar features) and (ii) intra-instance variations (viewpoint, pose, deformations, illumination, etc). Instead of combining two approaches with multi-task learning, we argue to organize and reason the data with multiple variations. Specifically, we propose to generate a graph with millions of objects mined from hundreds of thousands of videos. The objects are connected by two types of edges which correspond to two types of invariance: "different instances but a similar viewpoint and category" and "different viewpoints of the same instance". By applying simple transitivity on the graph with these edges, we can obtain pairs of images exhibiting richer visual invariance. We use this data to train a Triplet-Siamese network with VGG16 as the base architecture and apply the learned representations to different recognition tasks. For object detection, we achieve 63.2% mAP on PASCAL VOC 2007 using Fast R-CNN (compare to 67.3% with ImageNet pre-training). For the challenging COCO dataset, our method is surprisingly close (23.5%) to the ImageNet-supervised counterpart (24.4%) using the Faster R-CNN framework. We also show that our network can perform significantly better than the ImageNet network in the surface normal estimation task.

ICCV 2017

References in corpus (11)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Auto-Encoding Variational Bayes
• Unsupervised Learning of Video Representations using LSTMs
• Unsupervised Visual Representation Learning by Context Prediction
• Deep Convolutional Ranking for Multilabel Image Annotation
• Unsupervised Learning of Depth and Ego-Motion from Video
• Unsupervised Learning of Visual Representations using Videos
• Learning to See by Moving
• Learning Representations for Automatic Colorization
• Colorization as a Proxy Task for Visual Understanding
• Unsupervised Learning of Long-Term Motion Dynamics for Videos

Cited by in corpus (10)

• Deep Clustering for Unsupervised Learning of Visual Features
• Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination
• Unsupervised Learning of View-invariant Action Representations
• Learning Image Representations by Completing Damaged Jigsaw Puzzles
• Learning Video Object Segmentation from Unlabeled Videos
• Unsupervised learning from videos using temporal coherency deep networks
• Distilling Localization for Self-Supervised Representation Learning
• Self-Supervised Relative Depth Learning for Urban Scene Understanding
• Video Jigsaw: Unsupervised Learning of Spatiotemporal Context for Video Action Recognition
• Self-Supervised Visual Representations for Cross-Modal Retrieval