Watching the World Go By: Representation Learning from Unlabeled Videos

Abstract

Recent single image unsupervised representation learning techniques show remarkable success on a variety of tasks. The basic principle in these works is instance discrimination: learning to differentiate between two augmented versions of the same image and a large batch of unrelated images. Networks learn to ignore the augmentation noise and extract semantically meaningful representations. Prior work uses artificial data augmentation techniques such as cropping, and color jitter which can only affect the image in superficial ways and are not aligned with how objects actually change e.g. occlusion, deformation, viewpoint change. In this paper, we argue that videos offer this natural augmentation for free. Videos can provide entirely new views of objects, show deformation, and even connect semantically similar but visually distinct concepts. We propose Video Noise Contrastive Estimation, a method for using unlabeled video to learn strong, transferable single image representations. We demonstrate improvements over recent unsupervised single image techniques, as well as over fully supervised ImageNet pretraining, across a variety of temporal and non-temporal tasks. Code and the Random Related Video Views dataset are available at https://www.github.com/danielgordon10/vince

References in corpus (13)

• Adam: A Method for Stochastic Optimization
• Distributed Representations of Words and Phrases and their Compositionality
• A Simple Framework for Contrastive Learning of Visual Representations
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• The Kinetics Human Action Video Dataset
• Improved Baselines with Momentum Contrastive Learning
• GOT-10k: A Large High-Diversity Benchmark for Generic Object Tracking in the Wild
• Learning deep representations by mutual information estimation and maximization
• Unsupervised Learning of Video Representations using LSTMs
• Momentum Contrast for Unsupervised Visual Representation Learning
• YouTube-8M: A Large-Scale Video Classification Benchmark
• Data-Efficient Image Recognition with Contrastive Predictive Coding
• Contrastive Multiview Coding

Cited by in corpus (17)

• Contrastive Representation Learning: A Framework and Review
• What Makes for Good Views for Contrastive Learning?
• Space-Time Correspondence as a Contrastive Random Walk
• A Comprehensive Study of Deep Video Action Recognition
• Spatiotemporal Contrastive Video Representation Learning
• SODA10M: A Large-Scale 2D Self/Semi-Supervised Object Detection Dataset for Autonomous Driving
• A Large-Scale Study on Unsupervised Spatiotemporal Representation Learning
• Learning Video Representations from Textual Web Supervision
• Self-supervised Modal and View Invariant Feature Learning
• Back to the Future: Cycle Encoding Prediction for Self-supervised Contrastive Video Representation Learning
• Breaking Shortcut: Exploring Fully Convolutional Cycle-Consistency for Video Correspondence Learning
• What Can You Learn from Your Muscles? Learning Visual Representation from Human Interactions
• Video Contrastive Learning with Global Context
• The Impact of Spatiotemporal Augmentations on Self-Supervised Audiovisual Representation Learning
• Video Class Agnostic Segmentation with Contrastive Learning for Autonomous Driving
• Contrastive Learning of Image Representations with Cross-Video Cycle-Consistency
• PreViTS: Contrastive Pretraining with Video Tracking Supervision