A Meta-Transfer Objective for Learning to Disentangle Causal Mechanisms

Abstract

We propose to meta-learn causal structures based on how fast a learner adapts to new distributions arising from sparse distributional changes, e.g. due to interventions, actions of agents and other sources of non-stationarities. We show that under this assumption, the correct causal structural choices lead to faster adaptation to modified distributions because the changes are concentrated in one or just a few mechanisms when the learned knowledge is modularized appropriately. This leads to sparse expected gradients and a lower effective number of degrees of freedom needing to be relearned while adapting to the change. It motivates using the speed of adaptation to a modified distribution as a meta-learning objective. We demonstrate how this can be used to determine the cause-effect relationship between two observed variables. The distributional changes do not need to correspond to standard interventions (clamping a variable), and the learner has no direct knowledge of these interventions. We show that causal structures can be parameterized via continuous variables and learned end-to-end. We then explore how these ideas could be used to also learn an encoder that would map low-level observed variables to unobserved causal variables leading to faster adaptation out-of-distribution, learning a representation space where one can satisfy the assumptions of independent mechanisms and of small and sparse changes in these mechanisms due to actions and non-stationarities.

References in corpus (7)

• Learning Representations for Counterfactual Inference
• Invariant Models for Causal Transfer Learning
• The Consciousness Prior
• Domain Adaptation by Using Causal Inference to Predict Invariant Conditional Distributions
• Causal Reasoning from Meta-reinforcement Learning
• Nonlinear ICA Using Auxiliary Variables and Generalized Contrastive Learning
• Modular meta-learning

Cited by in corpus (59)

• Shortcut Learning in Deep Neural Networks
• Invariant Risk Minimization
• The Next Decade in AI: Four Steps Towards Robust Artificial Intelligence
• Logic Tensor Networks
• Causality for Machine Learning
• Preserving Causal Constraints in Counterfactual Explanations for Machine Learning Classifiers
• Weakly-Supervised Disentanglement Without Compromises
• Recurrent Independent Mechanisms
• A causal view of compositional zero-shot recognition
• Out of Distribution Generalization in Machine Learning
• On the Binding Problem in Artificial Neural Networks
• Learning Causal Semantic Representation for Out-of-Distribution Prediction
• Causal Induction from Visual Observations for Goal Directed Tasks
• Learning Neural Causal Models from Unknown Interventions
• Socially Responsible AI Algorithms: Issues, Purposes, and Challenges
• Meta-Learning Representations for Continual Learning
• Deep Stable Learning for Out-Of-Distribution Generalization
• Learning Compositional Neural Programs with Recursive Tree Search and Planning
• Learning Neural Causal Models with Active Interventions
• Disentangling User Interest and Conformity for Recommendation with Causal Embedding
• Can Subnetwork Structure be the Key to Out-of-Distribution Generalization?
• Learning Causal Models Online
• Out-of-Distribution Generalization Analysis via Influence Function
• A Review on Intelligent Object Perception Methods Combining Knowledge-based Reasoning and Machine Learning
• Variational Causal Networks: Approximate Bayesian Inference over Causal Structures
• Generalizing Graph Neural Networks on Out-Of-Distribution Graphs
• Size-Invariant Graph Representations for Graph Classification Extrapolations
• Dynamic Inference with Neural Interpreters
• S2RMs: Spatially Structured Recurrent Modules
• Meta-Learning Initializations for Image Segmentation
• Learning Temporally Causal Latent Processes from General Temporal Data
• Systematic Evaluation of Causal Discovery in Visual Model Based Reinforcement Learning
• Causal Attention for Vision-Language Tasks
• Re-using Adversarial Mask Discriminators for Test-time Training under Distribution Shifts
• A Bit More Bayesian: Domain-Invariant Learning with Uncertainty
• A Causal-based Framework for Multimodal Multivariate Time Series Validation Enhanced by Unsupervised Deep Learning as an Enabler for Industry 4.0
• A Combinatorial Perspective on Transfer Learning
• Amortized learning of neural causal representations
• Disentangling causal effects for hierarchical reinforcement learning
• Distinguishing rule- and exemplar-based generalization in learning systems
• Neural Function Modules with Sparse Arguments: A Dynamic Approach to Integrating Information across Layers
• Alleviating Privacy Attacks via Causal Learning
• Efficient Neural Causal Discovery without Acyclicity Constraints
• Review and Prospect: Deep Learning in Nuclear Magnetic Resonance Spectroscopy
• Compositional Attention: Disentangling Search and Retrieval
• On Hyper-parameter Tuning for Stochastic Optimization Algorithms
• Unaligned Image-to-Image Translation by Learning to Reweight
• The Effects of Invertibility on the Representational Complexity of Encoders in Variational Autoencoders
• Towards intervention-centric causal reasoning in learning agents
• Robustifying Sequential Neural Processes
• Understanding the Behaviour of the Empirical Cross-Entropy Beyond the Training Distribution
• A Meta Learning Approach to Discerning Causal Graph Structure
• Cycles in Causal Learning
• Causal Discovery by Kernel Intrinsic Invariance Measure
• On the Possibility of Rewarding Structure Learning Agents: Mutual Information on Linguistic Random Sets
• Exploiting non-i.i.d. data towards more robust machine learning algorithms
• Is Fast Adaptation All You Need?
• Causal Games and Causal Nash Equilibrium
• Signed Input Regularization