On Compositions of Transformations in Contrastive Self-Supervised Learning

Abstract

In the image domain, excellent representations can be learned by inducing invariance to content-preserving transformations via noise contrastive learning. In this paper, we generalize contrastive learning to a wider set of transformations, and their compositions, for which either invariance or distinctiveness is sought. We show that it is not immediately obvious how existing methods such as SimCLR can be extended to do so. Instead, we introduce a number of formal requirements that all contrastive formulations must satisfy, and propose a practical construction which satisfies these requirements. In order to maximise the reach of this analysis, we express all components of noise contrastive formulations as the choice of certain generalized transformations of the data (GDTs), including data sampling. We then consider videos as an example of data in which a large variety of transformations are applicable, accounting for the extra modalities -- for which we analyze audio and text -- and the dimension of time. We find that being invariant to certain transformations and distinctive to others is critical to learning effective video representations, improving the state-of-the-art for multiple benchmarks by a large margin, and even surpassing supervised pretraining.

arXiv commentAccepted to ICCV 2021. Code and pretrained models are available at https://github.com/facebookresearch/GDT

References in corpus (20)

• Efficient Estimation of Word Representations in Vector Space
• Representation Learning with Contrastive Predictive Coding
• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition
• The Kinetics Human Action Video Dataset
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
• Learning deep representations by mutual information estimation and maximization
• Data-Efficient Image Recognition with Contrastive Predictive Coding
• What Makes for Good Views for Contrastive Learning?
• Contrastive Multiview Coding
• Cooperative Learning of Audio and Video Models from Self-Supervised Synchronization
• Self-supervised Co-training for Video Representation Learning
• VideoBERT: A Joint Model for Video and Language Representation Learning
• Labelling unlabelled videos from scratch with multi-modal self-supervision
• Video Representation Learning with Visual Tempo Consistency
• Self-supervised Video Representation Learning by Pace Prediction
• Active Contrastive Learning of Audio-Visual Video Representations
• Learning Spatiotemporal Features via Video and Text Pair Discrimination
• Representation Learning with Video Deep InfoMax