Representation Learning via Invariant Causal Mechanisms

Abstract

Self-supervised learning has emerged as a strategy to reduce the reliance on costly supervised signal by pretraining representations only using unlabeled data. These methods combine heuristic proxy classification tasks with data augmentations and have achieved significant success, but our theoretical understanding of this success remains limited. In this paper we analyze self-supervised representation learning using a causal framework. We show how data augmentations can be more effectively utilized through explicit invariance constraints on the proxy classifiers employed during pretraining. Based on this, we propose a novel self-supervised objective, Representation Learning via Invariant Causal Mechanisms (ReLIC), that enforces invariant prediction of proxy targets across augmentations through an invariance regularizer which yields improved generalization guarantees. Further, using causality we generalize contrastive learning, a particular kind of self-supervised method, and provide an alternative theoretical explanation for the success of these methods. Empirically, ReLIC significantly outperforms competing methods in terms of robustness and out-of-distribution generalization on ImageNet, while also significantly outperforming these methods on Atari achieving above human-level performance on $51$ out of $57$ games.

References in corpus (16)

• A Simple Framework for Contrastive Learning of Visual Representations
• Bootstrap your own latent: A new approach to self-supervised Learning
• Improved Baselines with Momentum Contrastive Learning
• Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
• Learning deep representations by mutual information estimation and maximization
• Momentum Contrast for Unsupervised Visual Representation Learning
• Data-Efficient Image Recognition with Contrastive Predictive Coding
• What Makes for Good Views for Contrastive Learning?
• Contrastive Multiview Coding
• Large Batch Training of Convolutional Networks
• On Mutual Information Maximization for Representation Learning
• Learning Robust Global Representations by Penalizing Local Predictive Power
• A Theoretical Analysis of Contrastive Unsupervised Representation Learning
• On Variational Bounds of Mutual Information
• Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels
• Visual Causal Feature Learning

Cited by in corpus (18)

• Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges
• Predicting What You Already Know Helps: Provable Self-Supervised Learning
• Nonlinear Invariant Risk Minimization: A Causal Approach
• Desiderata for Representation Learning: A Causal Perspective
• Widening the Pipeline in Human-Guided Reinforcement Learning with Explanation and Context-Aware Data Augmentation
• Relating Graph Neural Networks to Structural Causal Models
• Counterfactual Learning on Graphs: A Survey
• Interpretable Water Level Forecaster with Spatiotemporal Causal Attention Mechanisms
• Distributional Anchor Regression
• A Theory of Label Propagation for Subpopulation Shift
• Counterfactual Invariance to Spurious Correlations: Why and How to Pass Stress Tests
• Causality-inspired Single-source Domain Generalization for Medical Image Segmentation
• Model-Invariant State Abstractions for Model-Based Reinforcement Learning
• Scaling Law for Recommendation Models: Towards General-purpose User Representations
• A Style and Semantic Memory Mechanism for Domain Generalization
• Self-Distilled Self-Supervised Representation Learning
• TumorCP: A Simple but Effective Object-Level Data Augmentation for Tumor Segmentation
• Generating Relevant and Coherent Dialogue Responses using Self-separated Conditional Variational AutoEncoders