Learning Causal Semantic Representation for Out-of-Distribution Prediction
arXiv:2011.01681
Abstract
Conventional supervised learning methods, especially deep ones, are found to be sensitive to out-of-distribution (OOD) examples, largely because the learned representation mixes the semantic factor with the variation factor due to their domain-specific correlation, while only the semantic factor causes the output. To address the problem, we propose a Causal Semantic Generative model (CSG) based on a causal reasoning so that the two factors are modeled separately, and develop methods for OOD prediction from a single training domain, which is common and challenging. The methods are based on the causal invariance principle, with a novel design in variational Bayes for both efficient learning and easy prediction. Theoretically, we prove that under certain conditions, CSG can identify the semantic factor by fitting training data, and this semantic-identification guarantees the boundedness of OOD generalization error and the success of adaptation. Empirical study shows improved OOD performance over prevailing baselines.
NeurIPS'21 camera-ready version
References in corpus (52)
- Adam: A Method for Stochastic Optimization
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- Auto-Encoding Variational Bayes
- Learning Transferable Features with Deep Adaptation Networks
- Semi-Supervised Learning with Deep Generative Models
- Conditional Adversarial Domain Adaptation
- Supervised Topic Models
- Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations
- Domain Generalization via Invariant Feature Representation
- Underspecification Presents Challenges for Credibility in Modern Machine Learning
- Adversarial Examples Are Not Bugs, They Are Features
- Generalizing Across Domains via Cross-Gradient Training
- Invariant Risk Minimization
- Towards a Definition of Disentangled Representations
- Causal Effect Inference with Deep Latent-Variable Models
- Out-of-Distribution Generalization via Risk Extrapolation (REx)
- On the Identifiability of the Post-Nonlinear Causal Model
- Bridging Theory and Algorithm for Domain Adaptation
- CAM: Causal additive models, high-dimensional order search and penalized regression
- How (not) to Train your Generative Model: Scheduled Sampling, Likelihood, Adversary?
- On Learning Invariant Representation for Domain Adaptation
- Invariant Models for Causal Transfer Learning
- Approximating CNNs with Bag-of-local-Features models works surprisingly well on ImageNet
- On Causal and Anticausal Learning
- Learning Disentangled Semantic Representation for Domain Adaptation
- A Meta-Transfer Objective for Learning to Disentangle Causal Mechanisms
- Variational Autoencoders and Nonlinear ICA: A Unifying Framework
- Causality for Machine Learning
- Causally Regularized Learning with Agnostic Data Selection Bias
- DIVA: Domain Invariant Variational Autoencoders
- In Search of Lost Domain Generalization
- Domain Adaptation by Using Causal Inference to Predict Invariant Conditional Distributions
- Towards Accurate Model Selection in Deep Unsupervised Domain Adaptation
- A causal view of compositional zero-shot recognition
- Weakly Supervised Disentanglement with Guarantees
- Anchor regression: heterogeneous data meets causality
- Disentangling Factors of Variation Using Few Labels
- Learning Neural Causal Models from Unknown Interventions
- Towards a Theoretical Framework of Out-of-Distribution Generalization
- Identifying confounders using additive noise models
- Few-shot Domain Adaptation by Causal Mechanism Transfer
- ICE-BeeM: Identifiable Conditional Energy-Based Deep Models Based on Nonlinear ICA
- Selecting Data Augmentation for Simulating Interventions
- Group invariance principles for causal generative models
- Generalization in anti-causal learning
- Causal Generative Domain Adaptation Networks
- CausalVAE: Structured Causal Disentanglement in Variational Autoencoder
- Detecting low-complexity unobserved causes
- Estimating Generalization under Distribution Shifts via Domain-Invariant Representations
- Identifying Finite Mixtures of Nonparametric Product Distributions and Causal Inference of Confounders
- Leveraging directed causal discovery to detect latent common causes
- Mitigating the Effects of Non-Identifiability on Inference for Bayesian Neural Networks with Latent Variables