Generalization in anti-causal learning

Abstract

The ability to learn and act in novel situations is still a prerogative of animate intelligence, as current machine learning methods mostly fail when moving beyond the standard i.i.d. setting. What is the reason for this discrepancy? Most machine learning tasks are anti-causal, i.e., we infer causes (labels) from effects (observations). Typically, in supervised learning we build systems that try to directly invert causal mechanisms. Instead, in this paper we argue that strong generalization capabilities crucially hinge on searching and validating meaningful hypotheses, requiring access to a causal model. In such a framework, we want to find a cause that leads to the observed effect. Anti-causal models are used to drive this search, but a causal model is required for validation. We investigate the fundamental differences between causal and anti-causal tasks, discuss implications for topics ranging from adversarial attacks to disentangling factors of variation, and provide extensive evidence from the literature to substantiate our view. We advocate for incorporating causal models in supervised learning to shift the paradigm from inference only, to search and validation.

arXiv commentA shorter version of this paper appeared at the workshop on `Critiquing and correcting trends in machine learning` at NeurIPS 2018

References in corpus (5)

• Properties of the Binary Black Hole Merger GW150914
• Attend, Infer, Repeat: Fast Scene Understanding with Generative Models
• DeepCoder: Learning to Write Programs
• Inferring deterministic causal relations
• Disentangling Factors of Variation via Generative Entangling

Cited by in corpus (13)

• Shortcut Learning in Deep Neural Networks
• Invariant Risk Minimization
• Causality for Machine Learning
• A causal view of compositional zero-shot recognition
• Learning Causal Semantic Representation for Out-of-Distribution Prediction
• A Causal View on Robustness of Neural Networks
• Desiderata for Representation Learning: A Causal Perspective
• Attribution-driven Causal Analysis for Detection of Adversarial Examples
• Nonuniform Negative Sampling and Log Odds Correction with Rare Events Data
• Adversarial Visual Robustness by Causal Intervention
• Visual Representation Learning Does Not Generalize Strongly Within the Same Domain
• Predicting with High Correlation Features
• Beyond traditional assumptions in fair machine learning