On Compositions of Transformations in Contrastive Self-Supervised Learning

Abstract

In the image domain, excellent representations can be learned by inducing invariance to content-preserving transformations via noise contrastive learning. In this paper, we generalize contrastive learning to a wider set of transformations, and their compositions, for which either invariance or distinctiveness is sought. We show that it is not immediately obvious how existing methods such as SimCLR can be extended to do so. Instead, we introduce a number of formal requirements that all contrastive formulations must satisfy, and propose a practical construction which satisfies these requirements. In order to maximise the reach of this analysis, we express all components of noise contrastive formulations as the choice of certain generalized transformations of the data (GDTs), including data sampling. We then consider videos as an example of data in which a large variety of transformations are applicable, accounting for the extra modalities -- for which we analyze audio and text -- and the dimension of time. We find that being invariant to certain transformations and distinctive to others is critical to learning effective video representations, improving the state-of-the-art for multiple benchmarks by a large margin, and even surpassing supervised pretraining.

Accepted to ICCV 2021. Code and pretrained models are available at https://github.com/facebookresearch/GDT

References in corpus (11)

• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• The Kinetics Human Action Video Dataset
• Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
• What Makes for Good Views for Contrastive Learning?
• Self-supervised Co-training for Video Representation Learning
• Labelling unlabelled videos from scratch with multi-modal self-supervision
• Video Representation Learning with Visual Tempo Consistency
• Self-supervised Video Representation Learning by Pace Prediction
• Active Contrastive Learning of Audio-Visual Video Representations
• Learning Spatiotemporal Features via Video and Text Pair Discrimination
• Representation Learning with Video Deep InfoMax