Convex Potential Flows: Universal Probability Distributions with Optimal Transport and Convex Optimization
arXiv:2012.05942
Abstract
Flow-based models are powerful tools for designing probabilistic models with tractable density. This paper introduces Convex Potential Flows (CP-Flow), a natural and efficient parameterization of invertible models inspired by the optimal transport (OT) theory. CP-Flows are the gradient map of a strongly convex neural potential function. The convexity implies invertibility and allows us to resort to convex optimization to solve the convex conjugate for efficient inversion. To enable maximum likelihood training, we derive a new gradient estimator of the log-determinant of the Jacobian, which involves solving an inverse-Hessian vector product using the conjugate gradient method. The gradient estimator has constant-memory cost, and can be made effectively unbiased by reducing the error tolerance level of the convex optimization routine. Theoretically, we prove that CP-Flows are universal density approximators and are optimal in the OT sense. Our empirical results show that CP-Flow performs competitively on standard benchmarks of density estimation and variational inference.
References in corpus (21)
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- The NumPy array: a structure for efficient numerical computation
- Spectral Normalization for Generative Adversarial Networks
- NICE: Non-linear Independent Components Estimation
- Variational Inference with Normalizing Flows
- Group Equivariant Convolutional Networks
- MADE: Masked Autoencoder for Distribution Estimation
- Input Convex Neural Networks
- Residual Flows for Invertible Generative Modeling
- Coupling-based Invertible Neural Networks Are Universal Diffeomorphism Approximators
- Monge-Ampère Flow for Generative Modeling
- Sum-of-Squares Polynomial Flow
- Optimal transport mapping via input convex neural networks
- Augmented Normalizing Flows: Bridging the Gap Between Generative Flows and Latent Variable Models
- 2-Wasserstein Approximation via Restricted Convex Potentials with Application to Improved Training for GANs
- Understanding and Mitigating Exploding Inverses in Invertible Neural Networks
- Wasserstein-2 Generative Networks
- Auto-Differentiating Linear Algebra
- Estimating the Spectral Density of Large Implicit Matrices
- Learning normalizing flows from Entropy-Kantorovich potentials
- Stochastic Chebyshev Gradient Descent for Spectral Optimization