Self-Supervised Learning with Data Augmentations Provably Isolates Content from Style

Abstract

Self-supervised representation learning has shown remarkable success in a number of domains. A common practice is to perform data augmentation via hand-crafted transformations intended to leave the semantics of the data invariant. We seek to understand the empirical success of this approach from a theoretical perspective. We formulate the augmentation process as a latent variable model by postulating a partition of the latent representation into a content component, which is assumed invariant to augmentation, and a style component, which is allowed to change. Unlike prior work on disentanglement and independent component analysis, we allow for both nontrivial statistical and causal dependencies in the latent space. We study the identifiability of the latent representation based on pairs of views of the observations and prove sufficient conditions that allow us to identify the invariant content partition up to an invertible mapping in both generative and discriminative settings. We find numerical simulations with dependent latent variables are consistent with our theory. Lastly, we introduce Causal3DIdent, a dataset of high-dimensional, visually complex images with rich causal dependencies, which we use to study the effect of data augmentations performed in practice.

NeurIPS 2021 final camera-ready revision (with minor corrections)

References in corpus (29)

• Distributed Representations of Words and Phrases and their Compositionality
• Auto-Encoding Variational Bayes
• Natural Language Processing (almost) from Scratch
• Improved Regularization of Convolutional Neural Networks with Cutout
• Shortcut Learning in Deep Neural Networks
• Learning deep representations by mutual information estimation and maximization
• NICE: Non-linear Independent Components Estimation
• Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations
• Disentangling by Factorising
• ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness
• An efficient framework for learning sentence representations
• Some Improvements on Deep Convolutional Neural Network Based Image Classification
• vq-wav2vec: Self-Supervised Learning of Discrete Speech Representations
• On Mutual Information Maximization for Representation Learning
• Understanding disentangling in $\beta$-VAE
• A Theoretical Analysis of Contrastive Unsupervised Representation Learning
• Unsupervised Feature Extraction by Time-Contrastive Learning and Nonlinear ICA
• Invariant Models for Causal Transfer Learning
• On Variational Bounds of Mutual Information
• Causality for Machine Learning
• A Kernel Theory of Modern Data Augmentation
• Weakly Supervised Disentanglement with Guarantees
• On the Transfer of Inductive Bias from Simulation to the Real World: a New Disentanglement Dataset
• Predicting What You Already Know Helps: Provable Self-Supervised Learning
• Gradient Starvation: A Learning Proclivity in Neural Networks
• Nonlinear Invariant Risk Minimization: A Causal Approach
• Towards causal generative scene models via competition of experts
• Contrastive estimation reveals topic posterior information to linear models
• CausalVAE: Structured Causal Disentanglement in Variational Autoencoder

Cited by in corpus (10)

• Learning Disentangled Representations in the Imaging Domain
• Interpretable and intervenable ultrasonography-based machine learning models for pediatric appendicitis
• Self-Supervised Learning for Invariant Representations from Multi-Spectral and SAR Images
• Causality and Independence Enhancement for Biased Node Classification
• Robust image representations with counterfactual contrastive learning
• Causal Inference via Style Transfer for Out-of-distribution Generalisation
• Evolutionary Augmentation Policy Optimization for Self-supervised Learning
• Style Feature Extraction Using Contrastive Conditioned Variational Autoencoders with Mutual Information Constraints
• Revisiting Deep Generalized Canonical Correlation Analysis
• Contrastive Representation Learning with Trainable Augmentation Channel