Disentangling representations in Restricted Boltzmann Machines without adversaries

Abstract

A goal of unsupervised machine learning is to build representations of complex high-dimensional data, with simple relations to their properties. Such disentangled representations make easier to interpret the significant latent factors of variation in the data, as well as to generate new data with desirable features. Methods for disentangling representations often rely on an adversarial scheme, in which representations are tuned to avoid discriminators from being able to reconstruct information about the data properties (labels). Unfortunately adversarial training is generally difficult to implement in practice. Here we propose a simple, effective way of disentangling representations without any need to train adversarial discriminators, and apply our approach to Restricted Boltzmann Machines (RBM), one of the simplest representation-based generative models. Our approach relies on the introduction of adequate constraints on the weights during training, which allows us to concentrate information about labels on a small subset of latent variables. The effectiveness of the approach is illustrated with four examples: the CelebA dataset of facial images, the two-dimensional Ising model, the MNIST dataset of handwritten digits, and the taxonomy of protein families. In addition, we show how our framework allows for analytically computing the cost, in terms of log-likelihood of the data, associated to the disentanglement of their representations.

arXiv commentMinor corrections. Accepted for publication in Physical Review X

References in corpus (12)

• Direct-coupling analysis of residue co-evolution captures native contacts across many protein families
• Influence of Generic Scale Invariance on Classical and Quantum Phase Transitions
• Inverse Statistical Physics of Protein Sequences: A Key Issues Review
• Emergence of Compositional Representations in Restricted Boltzmann Machines
• Critical Binder cumulant of two-dimensional Ising models
• Selection of sequence motifs and generative Hopfield-Potts models for protein familiesilies
• Machine learning determination of dynamical parameters: The Ising model case
• Equilibrium and non-Equilibrium regimes in the learning of Restricted Boltzmann Machines
• Transforming Generalized Ising Model into Boltzmann Machine
• Landau Theory for Non-Equilibrium Steady States
• 'Place-cell' emergence and learning of invariant data with restricted Boltzmann machines: breaking and dynamical restoration of continuous symmetries in the weight space
• Non-Analytic Non-Equilibrium Field Theory: Stochastic Reheating of the Ising Model