Rectangular Flows for Manifold Learning
arXiv:2106.01413
Abstract
Normalizing flows are invertible neural networks with tractable change-of-volume terms, which allow optimization of their parameters to be efficiently performed via maximum likelihood. However, data of interest are typically assumed to live in some (often unknown) low-dimensional manifold embedded in a high-dimensional ambient space. The result is a modelling mismatch since -- by construction -- the invertibility requirement implies high-dimensional support of the learned distribution. Injective flows, mappings from low- to high-dimensional spaces, aim to fix this discrepancy by learning distributions on manifolds, but the resulting volume-change term becomes more challenging to evaluate. Current approaches either avoid computing this term entirely using various heuristics, or assume the manifold is known beforehand and therefore are not widely applicable. Instead, we propose two methods to tractably calculate the gradient of this term with respect to the parameters of the model, relying on careful use of automatic differentiation and techniques from numerical linear algebra. Both approaches perform end-to-end nonlinear manifold learning and density estimation for data projected onto this manifold. We study the trade-offs between our proposed methods, empirically verify that we outperform approaches ignoring the volume-change term by more accurately learning manifolds and the corresponding distributions on them, and show promising results on out-of-distribution detection. Our code is available at https://github.com/layer6ai-labs/rectangular-flows.
arXiv commentNeurIPS 2021 Camera Ready. Code available at https://github.com/layer6ai-labs/rectangular-flows
References in corpus (20)
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- Auto-Encoding Variational Bayes
- Decoupled Weight Decay Regularization
- Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
- GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
- NICE: Non-linear Independent Components Estimation
- Variational Inference with Normalizing Flows
- Normalizing Flows: An Introduction and Review of Current Methods
- GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Acceleration
- Normalizing Flows for Probabilistic Modeling and Inference
- Residual Flows for Invertible Generative Modeling
- WAIC, but Why? Generative Ensembles for Robust Anomaly Detection
- Uncertainty in the Variational Information Bottleneck
- Do Deep Generative Models Know What They Don't Know?
- Normalizing Flows on Riemannian Manifolds
- Large-scale Log-determinant Computation through Stochastic Chebyshev Expansions
- Normalizing Flows Across Dimensions
- Implicit Normalizing Flows
- Tractable Density Estimation on Learned Manifolds with Conformal Embedding Flows