Unsupervised Learning from Video with Deep Neural Embeddings

Abstract

Because of the rich dynamical structure of videos and their ubiquity in everyday life, it is a natural idea that video data could serve as a powerful unsupervised learning signal for training visual representations in deep neural networks. However, instantiating this idea, especially at large scale, has remained a significant artificial intelligence challenge. Here we present the Video Instance Embedding (VIE) framework, which extends powerful recent unsupervised loss functions for learning deep nonlinear embeddings to multi-stream temporal processing architectures on large-scale video datasets. We show that VIE-trained networks substantially advance the state of the art in unsupervised learning from video datastreams, both for action recognition in the Kinetics dataset, and object recognition in the ImageNet dataset. We show that a hybrid model with both static and dynamic processing pathways is optimal for both transfer tasks, and provide analyses indicating how the pathways differ. Taken in context, our results suggest that deep neural embeddings are a promising approach to unsupervised visual learning across a wide variety of domains.

To appear in CVPR 2020

References in corpus (15)

• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• The Kinetics Human Action Video Dataset
• Learning deep representations by mutual information estimation and maximization
• Unsupervised Representation Learning by Predicting Image Rotations
• YouTube-8M: A Large-Scale Video Classification Benchmark
• Data-Efficient Image Recognition with Contrastive Predictive Coding
• Learning Representations by Maximizing Mutual Information Across Views
• Contrastive Multiview Coding
• Deep Predictive Coding Networks for Video Prediction and Unsupervised Learning
• PredRNN++: Towards A Resolution of the Deep-in-Time Dilemma in Spatiotemporal Predictive Learning
• SlowFast Networks for Video Recognition
• Self-Supervised Spatiotemporal Feature Learning via Video Rotation Prediction
• Task-Driven Convolutional Recurrent Models of the Visual System
• Learning Spatio-Temporal Features with 3D Residual Networks for Action Recognition
• Local Aggregation for Unsupervised Learning of Visual Embeddings

Cited by in corpus (5)

• What Makes for Good Views for Contrastive Learning?
• Contrastive Multiview Coding
• TCLR: Temporal Contrastive Learning for Video Representation
• Are we done with ImageNet?
• Long Short View Feature Decomposition via Contrastive Video Representation Learning