Improving Transformation Invariance in Contrastive Representation Learning

Abstract

We propose methods to strengthen the invariance properties of representations obtained by contrastive learning. While existing approaches implicitly induce a degree of invariance as representations are learned, we look to more directly enforce invariance in the encoding process. To this end, we first introduce a training objective for contrastive learning that uses a novel regularizer to control how the representation changes under transformation. We show that representations trained with this objective perform better on downstream tasks and are more robust to the introduction of nuisance transformations at test time. Second, we propose a change to how test time representations are generated by introducing a feature averaging approach that combines encodings from multiple transformations of the original input, finding that this leads to across the board performance gains. Finally, we introduce the novel Spirograph dataset to explore our ideas in the context of a differentiable generative process with multiple downstream tasks, showing that our techniques for learning invariance are highly beneficial.

arXiv commentPublished as a conference paper at ICLR 2021

References in corpus (21)

• A Simple Framework for Contrastive Learning of Visual Representations
• Improved Baselines with Momentum Contrastive Learning
• Learning deep representations by mutual information estimation and maximization
• SGDR: Stochastic Gradient Descent with Warm Restarts
• Momentum Contrast for Unsupervised Visual Representation Learning
• Data-Efficient Image Recognition with Contrastive Predictive Coding
• What Makes for Good Views for Contrastive Learning?
• Group Equivariant Convolutional Networks
• Tensor field networks: Rotation- and translation-equivariant neural networks for 3D point clouds
• Contrastive Multiview Coding
• Large Batch Training of Convolutional Networks
• Big Self-Supervised Models are Strong Semi-Supervised Learners
• Fractional Max-Pooling
• Robust Large Margin Deep Neural Networks
• Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere
• Lipschitz-Margin Training: Scalable Certification of Perturbation Invariance for Deep Neural Networks
• Contrastive Training for Improved Out-of-Distribution Detection
• On Nesting Monte Carlo Estimators
• On the Benefits of Invariance in Neural Networks
• Deeper Interpretability of Deep Networks
• Certifiably Robust Variational Autoencoders

Cited by in corpus (5)

• Data augmentation in Bayesian neural networks and the cold posterior effect
• Mean Embeddings with Test-Time Data Augmentation for Ensembling of Representations
• Equivariant Imaging: Learning Beyond the Range Space
• On Contrastive Representations of Stochastic Processes
• Provably Strict Generalisation Benefit for Invariance in Kernel Methods