A causal view of compositional zero-shot recognition

Abstract

People easily recognize new visual categories that are new combinations of known components. This compositional generalization capacity is critical for learning in real-world domains like vision and language because the long tail of new combinations dominates the distribution. Unfortunately, learning systems struggle with compositional generalization because they often build on features that are correlated with class labels even if they are not "essential" for the class. This leads to consistent misclassification of samples from a new distribution, like new combinations of known components. Here we describe an approach for compositional generalization that builds on causal ideas. First, we describe compositional zero-shot learning from a causal perspective, and propose to view zero-shot inference as finding "which intervention caused the image?". Second, we present a causal-inspired embedding model that learns disentangled representations of elementary components of visual objects from correlated (confounded) training data. We evaluate this approach on two datasets for predicting new combinations of attribute-object pairs: A well-controlled synthesized images dataset and a real-world dataset which consists of fine-grained types of shoes. We show improvements compared to strong baselines.

arXiv comment(1) Accepted to NeurIPS 2020 (Spotlight) (2) Project page is at https://github.com/nv-research-israel/causal_comp (3) A video of our spotlight talk is at https://www.youtube.com/watch?v=IUAmwBylvyc

References in corpus (21)

• Auto-Encoding Variational Bayes
• Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations
• Invariant Risk Minimization
• Causal Confusion in Imitation Learning
• On Causal and Anticausal Learning
• A Meta-Transfer Objective for Learning to Disentangle Causal Mechanisms
• Causality for Machine Learning
• Zero-Shot Learning -- The Good, the Bad and the Ugly
• An Empirical Study and Analysis of Generalized Zero-Shot Learning for Object Recognition in the Wild
• Anchor regression: heterogeneous data meets causality
• Learning to generalize to new compositions in image understanding
• C-VQA: A Compositional Split of the Visual Question Answering (VQA) v1.0 Dataset
• Improving One-Shot Learning through Fusing Side Information
• Learning Neural Causal Models from Unknown Interventions
• Probabilistic AND-OR Attribute Grouping for Zero-Shot Learning
• Generalization in anti-causal learning
• Deep Matching Autoencoders
• Learning Disentangled Representations via Mutual Information Estimation
• Robust Learning with the Hilbert-Schmidt Independence Criterion
• Conditional Dependence via Shannon Capacity: Axioms, Estimators and Applications
• Symmetry and Group in Attribute-Object Compositions

Cited by in corpus (6)

• Learning Causal Semantic Representation for Out-of-Distribution Prediction
• Using Deep Networks for Scientific Discovery in Physiological Signals
• CausalCity: Complex Simulations with Agency for Causal Discovery and Reasoning
• Causality-inspired Single-source Domain Generalization for Medical Image Segmentation
• Make an Omelette with Breaking Eggs: Zero-Shot Learning for Novel Attribute Synthesis
• Learning Single/Multi-Attribute of Object with Symmetry and Group