MIM: Mutual Information Machine
arXiv:1910.03175
Abstract
We introduce the Mutual Information Machine (MIM), a probabilistic auto-encoder for learning joint distributions over observations and latent variables. MIM reflects three design principles: 1) low divergence, to encourage the encoder and decoder to learn consistent factorizations of the same underlying distribution; 2) high mutual information, to encourage an informative relation between data and latent variables; and 3) low marginal entropy, or compression, which tends to encourage clustered latent representations. We show that a combination of the Jensen-Shannon divergence and the joint entropy of the encoding and decoding distributions satisfies these criteria, and admits a tractable cross-entropy bound that can be optimized directly with Monte Carlo and stochastic gradient descent. We contrast MIM learning with maximum likelihood and VAEs. Experiments show that MIM learns representations with high mutual information, consistent encoding and decoding distributions, effective latent clustering, and data log likelihood comparable to VAE, while avoiding posterior collapse.
Pre-print. Project webpage: https://research.seraphlabs.ca/projects/mim/
References in corpus (18)
- Adam: A Method for Stochastic Optimization
- Auto-Encoding Variational Bayes
- Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
- Estimating Mutual Information
- Conditional Image Generation with PixelCNN Decoders
- Learning deep representations by mutual information estimation and maximization
- NICE: Non-linear Independent Components Estimation
- Variational Inference with Normalizing Flows
- Density estimation using Real NVP
- Adversarial Feature Learning
- MINE: Mutual Information Neural Estimation
- InfoVAE: Information Maximizing Variational Autoencoders
- Variational Lossy Autoencoder
- Flow++: Improving Flow-Based Generative Models with Variational Dequantization and Architecture Design
- Improving Variational Inference with Inverse Autoregressive Flow
- REBAR: Low-variance, unbiased gradient estimates for discrete latent variable models
- VAE with a VampPrior
- The Information Autoencoding Family: A Lagrangian Perspective on Latent Variable Generative Models