Invariant Risk Minimization

Abstract

We introduce Invariant Risk Minimization (IRM), a learning paradigm to estimate invariant correlations across multiple training distributions. To achieve this goal, IRM learns a data representation such that the optimal classifier, on top of that data representation, matches for all training distributions. Through theory and experiments, we show how the invariances learned by IRM relate to the causal structures governing the data and enable out-of-distribution generalization.

References in corpus (12)

• ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness
• The Marginal Value of Adaptive Gradient Methods in Machine Learning
• Deep Learning: A Critical Appraisal
• Invariant Models for Causal Transfer Learning
• On Causal and Anticausal Learning
• A Meta-Transfer Objective for Learning to Disentangle Causal Mechanisms
• Learning to Pivot with Adversarial Networks
• Domain Adaptation by Using Causal Inference to Predict Invariant Conditional Distributions
• Intermittent process analysis with scattering moments
• Support and Invertibility in Domain-Invariant Representations
• Generalization in anti-causal learning
• From Dependence to Causation

Cited by in corpus (197)

• On the Opportunities and Risks of Foundation Models
• Shortcut Learning in Deep Neural Networks
• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• Underspecification Presents Challenges for Credibility in Modern Machine Learning
• Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization
• WILDS: A Benchmark of in-the-Wild Distribution Shifts
• Out-of-Distribution Generalization via Risk Extrapolation (REx)
• Towards Out-Of-Distribution Generalization: A Survey
• How Neural Networks Extrapolate: From Feedforward to Graph Neural Networks
• Preserving Causal Constraints in Counterfactual Explanations for Machine Learning Classifiers
• The Risks of Invariant Risk Minimization
• Generalizing to unseen domains via distribution matching
• In Search of Lost Domain Generalization
• Domain Generalization using Causal Matching
• A causal view of compositional zero-shot recognition
• Learning explanations that are hard to vary
• Domain Adaptation with Conditional Distribution Matching and Generalized Label Shift
• Efficient Domain Generalization via Common-Specific Low-Rank Decomposition
• Out of Distribution Generalization in Machine Learning
• Ensemble of Averages: Improving Model Selection and Boosting Performance in Domain Generalization
• Learning Causal Semantic Representation for Out-of-Distribution Prediction
• Causality in cognitive neuroscience: concepts, challenges, and distributional robustness
• Model Patching: Closing the Subgroup Performance Gap with Data Augmentation
• Invariance Principle Meets Information Bottleneck for Out-of-Distribution Generalization
• Subspace Regularizers for Few-Shot Class Incremental Learning
• Gradient Starvation: A Learning Proclivity in Neural Networks
• Towards a Theoretical Framework of Out-of-Distribution Generalization
• Unshuffling Data for Improved Generalization
• Few-shot Domain Adaptation by Causal Mechanism Transfer
• A Causal View on Robustness of Neural Networks
• Socially Responsible AI Algorithms: Issues, Purposes, and Challenges
• Invariant Risk Minimization Games
• When is invariance useful in an Out-of-Distribution Generalization problem ?
• Counterfactual Generative Networks
• Fairness via Representation Neutralization
• Improving out-of-distribution generalization via multi-task self-supervised pretraining
• Nonlinear Invariant Risk Minimization: A Causal Approach
• Evaluating Model Robustness and Stability to Dataset Shift
• Does Invariant Risk Minimization Capture Invariance?
• SAND-mask: An Enhanced Gradient Masking Strategy for the Discovery of Invariances in Domain Generalization
• Selecting Data Augmentation for Simulating Interventions
• Counterfactual Zero-Shot and Open-Set Visual Recognition
• Rational Shapley Values
• An Empirical Study of Invariant Risk Minimization
• Deep Stable Learning for Out-Of-Distribution Generalization
• Feature Expansive Reward Learning: Rethinking Human Input
• BREEDS: Benchmarks for Subpopulation Shift
• DoWhy: Addressing Challenges in Expressing and Validating Causal Assumptions
• Efficient nonparametric statistical inference on population feature importance using Shapley values
• Desiderata for Representation Learning: A Causal Perspective
• Observational Overfitting in Reinforcement Learning
• Out-of-distribution Prediction with Invariant Risk Minimization: The Limitation and An Effective Fix
• Can Subnetwork Structure be the Key to Out-of-Distribution Generalization?
• Empirical or Invariant Risk Minimization? A Sample Complexity Perspective
• Learning to Estimate Without Bias
• Causal Inference in Natural Language Processing: Estimation, Prediction, Interpretation and Beyond
• Gradient Matching for Domain Generalization
• Saliency is a Possible Red Herring When Diagnosing Poor Generalization
• Domain adaptation under structural causal models
• Learning Causal Models Online
• Linear unit-tests for invariance discovery
• Invariant Policy Optimization: Towards Stronger Generalization in Reinforcement Learning
• WQT and DG-YOLO: towards domain generalization in underwater object detection
• Variational Disentanglement for Domain Generalization
• Invariant Representation Learning for Treatment Effect Estimation
• Representation via Representations: Domain Generalization via Adversarially Learned Invariant Representations
• Learning Invariant Representations and Risks for Semi-supervised Domain Adaptation
• Algorithms for Causal Reasoning in Probability Trees
• Domain Invariant Representation Learning with Domain Density Transformations
• On Calibration and Out-of-domain Generalization
• Partial Optimal Transport with Applications on Positive-Unlabeled Learning
• Out-of-Distribution Generalization Analysis via Influence Function
• On the Transfer of Disentangled Representations in Realistic Settings
• Robust Learning with the Hilbert-Schmidt Independence Criterion
• Latent Causal Invariant Model
• Continual Reinforcement Learning with Multi-Timescale Replay
• Generalizing Graph Neural Networks on Out-Of-Distribution Graphs
• An Empirical Study on Robustness to Spurious Correlations using Pre-trained Language Models
• Algorithmic Bias and Data Bias: Understanding the Relation between Distributionally Robust Optimization and Data Curation
• Think before you act: A simple baseline for compositional generalization
• Size-Invariant Graph Representations for Graph Classification Extrapolations
• Scene Transformer: A unified architecture for predicting multiple agent trajectories
• Latent Adversarial Debiasing: Mitigating Collider Bias in Deep Neural Networks
• In-N-Out: Pre-Training and Self-Training using Auxiliary Information for Out-of-Distribution Robustness
• Overinterpretation reveals image classification model pathologies
• Modular Meta-Learning with Shrinkage
• Causally motivated Shortcut Removal Using Auxiliary Labels
• Fairness and Robustness in Invariant Learning: A Case Study in Toxicity Classification
• Distributional Anchor Regression
• Intervention Design for Effective Sim2Real Transfer
• A Theory of Label Propagation for Subpopulation Shift
• Proactive Pseudo-Intervention: Causally Informed Contrastive Learning For Interpretable Vision Models
• Enforcing Predictive Invariance across Structured Biomedical Domains
• Counterfactual Invariance to Spurious Correlations: Why and How to Pass Stress Tests
• Shaking the foundations: delusions in sequence models for interaction and control
• Evaluation of Complexity Measures for Deep Learning Generalization in Medical Image Analysis
• I-SPEC: An End-to-End Framework for Learning Transportable, Shift-Stable Models
• Continuous Coordination As a Realistic Scenario for Lifelong Learning
• Searching for consistent associations with a multi-environment knockoff filter
• Explaining The Efficacy of Counterfactually Augmented Data
• Training for the Future: A Simple Gradient Interpolation Loss to Generalize Along Time
• Quantifying and Improving Transferability in Domain Generalization
• Accounting for Unobserved Confounding in Domain Generalization
• Causality-aware counterfactual confounding adjustment for feature representations learned by deep models
• Causality-aware counterfactual confounding adjustment as an alternative to linear residualization in anticausal prediction tasks based on linear learners
• Causal Analysis of Agent Behavior for AI Safety
• Visual Representation Learning Does Not Generalize Strongly Within the Same Domain
• MUTANT: A Training Paradigm for Out-of-Distribution Generalization in Visual Question Answering
• An Online Learning Approach to Interpolation and Extrapolation in Domain Generalization
• Model-Based Domain Generalization
• Understanding Continual Learning Settings with Data Distribution Drift Analysis
• Rearchitecting Classification Frameworks For Increased Robustness
• Log-Likelihood Ratio Minimizing Flows: Towards Robust and Quantifiable Neural Distribution Alignment
• Understanding and Testing Generalization of Deep Networks on Out-of-Distribution Data
• A Fine-Grained Analysis on Distribution Shift
• Iterative Feature Matching: Toward Provable Domain Generalization with Logarithmic Environments
• SWAD: Domain Generalization by Seeking Flat Minima
• In Search of Robust Measures of Generalization
• Adaptive Invariance for Molecule Property Prediction
• How Useful Are the Machine-Generated Interpretations to General Users? A Human Evaluation on Guessing the Incorrectly Predicted Labels
• Towards Principled Disentanglement for Domain Generalization
• Meta-Learned Invariant Risk Minimization
• Decoupling Value and Policy for Generalization in Reinforcement Learning
• Linear Regression Games: Convergence Guarantees to Approximate Out-of-Distribution Solutions
• An Information-theoretic Approach to Distribution Shifts
• Predicting with High Correlation Features
• Towards Causal Federated Learning For Enhanced Robustness and Privacy
• The Connection between Out-of-Distribution Generalization and Privacy of ML Models
• Can You Learn an Algorithm? Generalizing from Easy to Hard Problems with Recurrent Networks
• Multi-label Iterated Learning for Image Classification with Label Ambiguity
• Causality-inspired Single-source Domain Generalization for Medical Image Segmentation
• Robustness through Data Augmentation Loss Consistency
• Distinguishing rule- and exemplar-based generalization in learning systems
• Focus on the Common Good: Group Distributional Robustness Follows
• Retiring Adult: New Datasets for Fair Machine Learning
• Source-Free Adaptation to Measurement Shift via Bottom-Up Feature Restoration
• Defending Against Image Corruptions Through Adversarial Augmentations
• Model-Invariant State Abstractions for Model-Based Reinforcement Learning
• Identifying Invariant Factors Across Multiple Environments with KL Regression
• Alleviating Privacy Attacks via Causal Learning
• Robust Representation Learning via Perceptual Similarity Metrics
• Embracing the Disharmony in Medical Imaging: A Simple and Effective Framework for Domain Adaptation
• Representation Bayesian Risk Decompositions and Multi-Source Domain Adaptation
• Embracing the Dark Knowledge: Domain Generalization Using Regularized Knowledge Distillation
• Online Safety Assurance for Deep Reinforcement Learning
• Uniform Priors for Data-Efficient Transfer
• Harmonization with Flow-based Causal Inference
• Causal Attention for Unbiased Visual Recognition
• Optimization-based Causal Estimation from Heterogenous Environments
• Target Consistency for Domain Adaptation: when Robustness meets Transferability
• Variance Minimization in the Wasserstein Space for Invariant Causal Prediction
• Causal bootstrapping
• Learning Diverse Representations for Fast Adaptation to Distribution Shift
• For Manifold Learning, Deep Neural Networks can be Locality Sensitive Hash Functions
• Rotation-Invariant Autoencoders for Signals on Spheres
• Predicting the Accuracy of a Few-Shot Classifier
• Stochastic Adversarial Gradient Embedding for Active Domain Adaptation
• Learning Stable Classifiers by Transferring Unstable Features
• Balance Regularized Neural Network Models for Causal Effect Estimation
• Towards Robust and Adaptive Motion Forecasting: A Causal Representation Perspective
• Predict then Interpolate: A Simple Algorithm to Learn Stable Classifiers
• Regularizing towards Causal Invariance: Linear Models with Proxies
• Causal Learning for Socially Responsible AI
• Independent mechanism analysis, a new concept?
• Encouraging Intra-Class Diversity Through a Reverse Contrastive Loss for Better Single-Source Domain Generalization
• TAMPC: A Controller for Escaping Traps in Novel Environments
• FOCUS: Familiar Objects in Common and Uncommon Settings
• Contrastive ACE: Domain Generalization Through Alignment of Causal Mechanisms
• On Invariance Penalties for Risk Minimization
• Generalizing Nucleus Recognition Model in Multi-source Images via Pruning
• Interventional Domain Adaptation
• Stable predictions for health related anticausal prediction tasks affected by selection biases: the need to deconfound the test set features
• Detecting Spurious Correlations with Sanity Tests for Artificial Intelligence Guided Radiology Systems
• Robust Classification under Class-Dependent Domain Shift
• Learning Under Adversarial and Interventional Shifts
• Neural Networks for Learning Counterfactual G-Invariances from Single Environments
• Invariant Risk Minimisation for Cross-Organism Inference: Substituting Mouse Data for Human Data in Human Risk Factor Discovery
• Causal Transfer Random Forest: Combining Logged Data and Randomized Experiments for Robust Prediction
• Selective Regression Under Fairness Criteria
• Knowledge as Invariance -- History and Perspectives of Knowledge-augmented Machine Learning
• Distributionally Robust Learning with Stable Adversarial Training
• Finding Valid Adjustments under Non-ignorability with Minimal DAG Knowledge
• Combining Diverse Feature Priors
• Unsupervised Recalibration
• An Empirical Framework for Domain Generalization in Clinical Settings
• Robust Learning in Heterogeneous Contexts
• Gradient Masked Federated Optimization
• Towards intervention-centric causal reasoning in learning agents
• Reappraising Domain Generalization in Neural Networks
• Evaluation of Causal Structure Learning Algorithms via Risk Estimation
• Effects of Multi-Aspect Online Reviews with Unobserved Confounders: Estimation and Implication
• Task Guided Compositional Representation Learning for ZDA
• Total Recall: a Customized Continual Learning Method for Neural Semantic Parsers
• Bandits Don't Follow Rules: Balancing Multi-Facet Machine Translation with Multi-Armed Bandits
• Beyond Discriminant Patterns: On the Robustness of Decision Rule Ensembles
• Domain Conditional Predictors for Domain Adaptation
• Learning Invariants through Soft Unification