Labelling unlabelled videos from scratch with multi-modal self-supervision

Abstract

A large part of the current success of deep learning lies in the effectiveness of data -- more precisely: labelled data. Yet, labelling a dataset with human annotation continues to carry high costs, especially for videos. While in the image domain, recent methods have allowed to generate meaningful (pseudo-) labels for unlabelled datasets without supervision, this development is missing for the video domain where learning feature representations is the current focus. In this work, we a) show that unsupervised labelling of a video dataset does not come for free from strong feature encoders and b) propose a novel clustering method that allows pseudo-labelling of a video dataset without any human annotations, by leveraging the natural correspondence between the audio and visual modalities. An extensive analysis shows that the resulting clusters have high semantic overlap to ground truth human labels. We further introduce the first benchmarking results on unsupervised labelling of common video datasets Kinetics, Kinetics-Sound, VGG-Sound and AVE.

Accepted to NeurIPS 2020. Project page: https://www.robots.ox.ac.uk/~vgg/research/selavi, code: https://github.com/facebookresearch/selavi

References in corpus (19)

• Distilling the Knowledge in a Neural Network
• A Simple Framework for Contrastive Learning of Visual Representations
• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• The Kinetics Human Action Video Dataset
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• YouTube-8M: A Large-Scale Video Classification Benchmark
• Deep Image Prior
• Contrastive Multiview Coding
• Prototypical Contrastive Learning of Unsupervised Representations
• Cooperative Learning of Audio and Video Models from Self-Supervised Synchronization
• Learning Discrete Representations via Information Maximizing Self-Augmented Training
• Self-Supervised Learning by Cross-Modal Audio-Video Clustering
• Learning Video Representations using Contrastive Bidirectional Transformer
• Audiovisual SlowFast Networks for Video Recognition
• Self-Supervised Spatiotemporal Feature Learning via Video Rotation Prediction
• A critical analysis of self-supervision, or what we can learn from a single image
• Learning Spatiotemporal Features via Video and Text Pair Discrimination
• Self-Supervised Visual Learning by Variable Playback Speeds Prediction of a Video
• Deep Amortized Clustering

Cited by in corpus (13)

• Hard Negative Mixing for Contrastive Learning
• Self-Supervised Learning for Videos: A Survey
• Spatiotemporal Contrastive Video Representation Learning
• Active Contrastive Learning of Audio-Visual Video Representations
• Support-set bottlenecks for video-text representation learning
• Revisiting Contrastive Methods for Unsupervised Learning of Visual Representations
• Overcoming Failures of Imagination in AI Infused System Development and Deployment
• Revitalizing CNN Attentions via Transformers in Self-Supervised Visual Representation Learning
• Induction Network: Audio-Visual Modality Gap-Bridging for Self-Supervised Sound Source Localization
• Contrastive Learning of Global-Local Video Representations
• Joint Representation Learning and Novel Category Discovery on Single- and Multi-modal Data
• Overcoming the Domain Gap in Contrastive Learning of Neural Action Representations
• On Compositions of Transformations in Contrastive Self-Supervised Learning