Statistically Optimal Generative Modeling with Maximum Deviation from the Empirical Distribution

Abstract

This paper explores the problem of generative modeling, aiming to simulate diverse examples from an unknown distribution based on observed examples. While recent studies have focused on quantifying the statistical precision of popular algorithms, there is a lack of mathematical evaluation regarding the non-replication of observed examples and the creativity of the generative model. We present theoretical insights into this aspect, demonstrating that the Wasserstein GAN, constrained to left-invertible push-forward maps, generates distributions that avoid replication and significantly deviate from the empirical distribution. Importantly, we show that left-invertibility achieves this without compromising the statistical optimality of the resulting generator. Our most important contribution provides a finite-sample lower bound on the Wasserstein-1 distance between the generative distribution and the empirical one. We also establish a finite-sample upper bound on the distance between the generative distribution and the true data-generating one. Both bounds are explicit and show the impact of key parameters such as sample size, dimensions of the ambient and latent spaces, noise level, and smoothness measured by the Lipschitz constant.

ICML 2024

References in corpus (20)

• Generative Adversarial Networks
• Extracting Training Data from Diffusion Models
• Towards GAN Benchmarks Which Require Generalization
• Riemannian Score-Based Generative Modelling
• Understanding and Mitigating Copying in Diffusion Models
• Some Theoretical Insights into Wasserstein GANs
• An error analysis of generative adversarial networks for learning distributions
• Measuring Forgetting of Memorized Training Examples
• Statistical guarantees for generative models without domination
• A likelihood approach to nonparametric estimation of a singular distribution using deep generative models
• Generalization Properties of Optimal Transport GANs with Latent Distribution Learning
• Ambient Diffusion: Learning Clean Distributions from Corrupted Data
• Rates of convergence for density estimation with generative adversarial networks
• Wasserstein GANs are Minimax Optimal Distribution Estimators
• Minimax Rate of Distribution Estimation on Unknown Submanifold under Adversarial Losses
• Direct Parameterization of Lipschitz-Bounded Deep Networks
• A Good Score Does not Lead to A Good Generative Model
• Minimax Optimality (Probably) Doesn't Imply Distribution Learning for GANs
• Indeterminacy in Generative Models: Characterization and Strong Identifiability
• Minimax optimal density estimation using a shallow generative model with a one-dimensional latent variable