On Linear Identifiability of Learned Representations

Abstract

Identifiability is a desirable property of a statistical model: it implies that the true model parameters may be estimated to any desired precision, given sufficient computational resources and data. We study identifiability in the context of representation learning: discovering nonlinear data representations that are optimal with respect to some downstream task. When parameterized as deep neural networks, such representation functions typically lack identifiability in parameter space, because they are overparameterized by design. In this paper, building on recent advances in nonlinear ICA, we aim to rehabilitate identifiability by showing that a large family of discriminative models are in fact identifiable in function space, up to a linear indeterminacy. Many models for representation learning in a wide variety of domains have been identifiable in this sense, including text, images and audio, state-of-the-art at time of publication. We derive sufficient conditions for linear identifiability and provide empirical support for the result on both simulated and real-world data.

References in corpus (10)

• Distributed Representations of Words and Phrases and their Compositionality
• Language Models are Few-Shot Learners
• XLNet: Generalized Autoregressive Pretraining for Language Understanding
• Data-Efficient Image Recognition with Contrastive Predictive Coding
• Semi-supervised Sequence Learning
• SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability
• Variational Autoencoders and Nonlinear ICA: A Unifying Framework
• Disentanglement by Nonlinear ICA with General Incompressible-flow Networks (GIN)
• Nonlinear ICA Using Auxiliary Variables and Generalized Contrastive Learning
• ICE-BeeM: Identifiable Conditional Energy-Based Deep Models Based on Nonlinear ICA

Cited by in corpus (8)

• Score-Based Generative Modeling through Stochastic Differential Equations
• Contrastive Learning Inverts the Data Generating Process
• Repulsive Deep Ensembles are Bayesian
• Unsupervised Learning of Compositional Energy Concepts
• Visual Representation Learning Does Not Generalize Strongly Within the Same Domain
• Intact-VAE: Estimating Treatment Effects under Unobserved Confounding
• Exploring the Interchangeability of CNN Embedding Spaces
• Video Reenactment as Inductive Bias for Content-Motion Disentanglement