The Information Autoencoding Family: A Lagrangian Perspective on Latent Variable Generative Models

Abstract

A large number of objectives have been proposed to train latent variable generative models. We show that many of them are Lagrangian dual functions of the same primal optimization problem. The primal problem optimizes the mutual information between latent and visible variables, subject to the constraints of accurately modeling the data distribution and performing correct amortized inference. By choosing to maximize or minimize mutual information, and choosing different Lagrange multipliers, we obtain different objectives including InfoGAN, ALI/BiGAN, ALICE, CycleGAN, beta-VAE, adversarial autoencoders, AVB, AS-VAE and InfoVAE. Based on this observation, we provide an exhaustive characterization of the statistical and computational trade-offs made by all the training objectives in this class of Lagrangian duals. Next, we propose a dual optimization method where we optimize model parameters as well as the Lagrange multipliers. This method achieves Pareto optimal solutions in terms of optimizing information and satisfying the constraints.

References in corpus (18)

• Adam: A Method for Stochastic Optimization
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• Practical Bayesian Optimization of Machine Learning Algorithms
• Categorical Reparameterization with Gumbel-Softmax
• Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks
• InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets
• Adversarial Feature Learning
• Adversarially Learned Inference
• f-GAN: Training Generative Neural Samplers using Variational Divergence Minimization
• Generative Moment Matching Networks
• InfoVAE: Information Maximizing Variational Autoencoders
• Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithm
• A Kernel Method for the Two-Sample Problem
• On Convergence and Stability of GANs
• Wasserstein Auto-Encoders
• Learning in Implicit Generative Models
• Flow-GAN: Combining Maximum Likelihood and Adversarial Learning in Generative Models
• Modeling Sparse Deviations for Compressed Sensing using Generative Models

Cited by in corpus (16)

• IB-GAN: Disentangled Representation Learning with Information Bottleneck Generative Adversarial Networks
• A Convex Duality Framework for GANs
• Understanding the Limitations of Variational Mutual Information Estimators
• Information bottleneck through variational glasses
• Constrained Learning with Non-Convex Losses
• Learnable Bernoulli Dropout for Bayesian Deep Learning
• Deep Evidential Regression
• How to Hallucinate Functional Proteins
• Physics-Integrated Variational Autoencoders for Robust and Interpretable Generative Modeling
• DEMI: Discriminative Estimator of Mutual Information
• Select and Attend: Towards Controllable Content Selection in Text Generation
• Infomax Neural Joint Source-Channel Coding via Adversarial Bit Flip
• Probably Approximately Correct Constrained Learning
• MIM: Mutual Information Machine
• Inferential Wasserstein Generative Adversarial Networks
• Characterizing and Avoiding Problematic Global Optima of Variational Autoencoders