Statistically Optimal Generative Modeling with Maximum Deviation from the Empirical Distribution

Abstract

This paper explores the problem of generative modeling, aiming to simulate diverse examples from an unknown distribution based on observed examples. While recent studies have focused on quantifying the statistical precision of popular algorithms, there is a lack of mathematical evaluation regarding the non-replication of observed examples and the creativity of the generative model. We present theoretical insights into this aspect, demonstrating that the Wasserstein GAN, constrained to left-invertible push-forward maps, generates distributions that avoid replication and significantly deviate from the empirical distribution. Importantly, we show that left-invertibility achieves this without compromising the statistical optimality of the resulting generator. Our most important contribution provides a finite-sample lower bound on the Wasserstein-1 distance between the generative distribution and the empirical one. We also establish a finite-sample upper bound on the distance between the generative distribution and the true data-generating one. Both bounds are explicit and show the impact of key parameters such as sample size, dimensions of the ambient and latent spaces, noise level, and smoothness measured by the Lipschitz constant.

ICML 2024

References in corpus (27)

• Generative Adversarial Networks
• VEEGAN: Reducing Mode Collapse in GANs using Implicit Variational Learning
• CaloGAN: Simulating 3D High Energy Particle Showers in Multi-Layer Electromagnetic Calorimeters with Generative Adversarial Networks
• Machine Learning for Stochastic Parameterization: Generative Adversarial Networks in the Lorenz '96 Model
• Stabilizing Training of Generative Adversarial Networks through Regularization
• On the regularization of Wasserstein GANs
• Extracting Training Data from Diffusion Models
• BourGAN: Generative Networks with Metric Embeddings
• Towards GAN Benchmarks Which Require Generalization
• Exactly Computing the Local Lipschitz Constant of ReLU Networks
• Riemannian Score-Based Generative Modelling
• Some Theoretical Insights into Wasserstein GANs
• Understanding and Mitigating Copying in Diffusion Models
• An error analysis of generative adversarial networks for learning distributions
• Measuring Forgetting of Memorized Training Examples
• A likelihood approach to nonparametric estimation of a singular distribution using deep generative models
• Statistical guarantees for generative models without domination
• Generalization Properties of Optimal Transport GANs with Latent Distribution Learning
• Ambient Diffusion: Learning Clean Distributions from Corrupted Data
• Rates of convergence for density estimation with generative adversarial networks
• Minimax Rate of Distribution Estimation on Unknown Submanifold under Adversarial Losses
• Wasserstein GANs are Minimax Optimal Distribution Estimators
• Direct Parameterization of Lipschitz-Bounded Deep Networks
• Minimax optimal density estimation using a shallow generative model with a one-dimensional latent variable
• Indeterminacy in Generative Models: Characterization and Strong Identifiability
• Minimax Optimality (Probably) Doesn't Imply Distribution Learning for GANs
• A Good Score Does not Lead to A Good Generative Model