Desiderata for Representation Learning: A Causal Perspective
arXiv:2109.03795
Abstract
Representation learning constructs low-dimensional representations to summarize essential features of high-dimensional data. This learning problem is often approached by describing various desiderata associated with learned representations; e.g., that they be non-spurious, efficient, or disentangled. It can be challenging, however, to turn these intuitive desiderata into formal criteria that can be measured and enhanced based on observed data. In this paper, we take a causal perspective on representation learning, formalizing non-spuriousness and efficiency (in supervised representation learning) and disentanglement (in unsupervised representation learning) using counterfactual quantities and observable consequences of causal assertions. This yields computable metrics that can be used to assess the degree to which representations satisfy the desiderata of interest and learn non-spurious and disentangled representations from single observational datasets.
68 pages
References in corpus (9)
- Underspecification Presents Challenges for Credibility in Modern Machine Learning
- On Learning Invariant Representation for Domain Adaptation
- Multi-Level Variational Autoencoder: Learning Disentangled Representations from Grouped Observations
- Independently Controllable Factors
- Graph-based Isometry Invariant Representation Learning
- Using Embeddings to Correct for Unobserved Confounding in Networks
- Disentangled Representation Learning with Wasserstein Total Correlation
- Deep causal representation learning for unsupervised domain adaptation
- Counterfactual Invariance to Spurious Correlations: Why and How to Pass Stress Tests