Max-Affine Spline Insights into Deep Generative Networks

Abstract

We connect a large class of Generative Deep Networks (GDNs) with spline operators in order to derive their properties, limitations, and new opportunities. By characterizing the latent space partition, dimension and angularity of the generated manifold, we relate the manifold dimension and approximation error to the sample size. The manifold-per-region affine subspace defines a local coordinate basis; we provide necessary and sufficient conditions relating those basis vectors with disentanglement. We also derive the output probability density mapped onto the generated manifold in terms of the latent space density, which enables the computation of key statistics such as its Shannon entropy. This finding also enables the computation of the GDN likelihood, which provides a new mechanism for model comparison as well as providing a quality measure for (generated) samples under the learned distribution. We demonstrate how low entropy and/or multimodal distributions are not naturally modeled by DGNs and are a cause of training instabilities.

References in corpus (19)

• Auto-Encoding Variational Bayes
• Neural Discrete Representation Learning
• NICE: Non-linear Independent Components Estimation
• Variational Inference with Normalizing Flows
• Energy-based Generative Adversarial Network
• Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations
• Density estimation using Real NVP
• Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Data
• Dropout Training as Adaptive Regularization
• On Convergence and Stability of GANs
• Variational Recurrent Auto-Encoders
• Diversity-Sensitive Conditional Generative Adversarial Networks
• Sylvester Normalizing Flows for Variational Inference
• Hyperspherical Variational Auto-Encoders
• Improving Variational Auto-Encoders using Householder Flow
• On the Discrimination-Generalization Tradeoff in GANs
• Theory and Experiments on Vector Quantized Autoencoders
• Gaussian Mixture Generative Adversarial Networks for Diverse Datasets, and the Unsupervised Clustering of Images
• A RAD approach to deep mixture models

Cited by in corpus (4)

• Globally Injective ReLU Networks
• MaGNET: Uniform Sampling from Deep Generative Network Manifolds Without Retraining
• Analytical Probability Distributions and EM-Learning for Deep Generative Networks
• Fast Jacobian-Vector Product for Deep Networks