How Well Generative Adversarial Networks Learn Distributions

Abstract

This paper studies the rates of convergence for learning distributions implicitly with the adversarial framework and Generative Adversarial Networks (GANs), which subsume Wasserstein, Sobolev, MMD GAN, and Generalized/Simulated Method of Moments (GMM/SMM) as special cases. We study a wide range of parametric and nonparametric target distributions under a host of objective evaluation metrics. We investigate how to obtain valid statistical guarantees for GANs through the lens of regularization. On the nonparametric end, we derive the optimal minimax rates for distribution estimation under the adversarial framework. On the parametric end, we establish a theory for general neural network classes (including deep leaky ReLU networks) that characterizes the interplay on the choice of generator and discriminator pair. We discover and isolate a new notion of regularization, called the generator-discriminator-pair regularization, that sheds light on the advantage of GANs compared to classical parametric and nonparametric approaches for explicit distribution estimation. We develop novel oracle inequalities as the main technical tools for analyzing GANs, which are of independent interest.

Journal of Machine Learning Research, to appear

References in corpus (8)

• Generative Moment Matching Networks
• Towards Principled Methods for Training Generative Adversarial Networks
• Amortised MAP Inference for Image Super-resolution
• Do GANs actually learn the distribution? An empirical study
• Learning Probability Measures with respect to Optimal Transport Metrics
• The Inductive Bias of Restricted f-GANs
• SGD Learns One-Layer Networks in WGANs
• Estimating Certain Integral Probability Metric (IPM) is as Hard as Estimating under the IPM

Cited by in corpus (17)

• A Review on Generative Adversarial Networks: Algorithms, Theory, and Applications
• Wasserstein Generative Adversarial Uncertainty Quantification in Physics-Informed Neural Networks
• On the capacity of deep generative networks for approximating distributions
• An error analysis of generative adversarial networks for learning distributions
• Statistical guarantees for generative models without domination
• A likelihood approach to nonparametric estimation of a singular distribution using deep generative models
• SGD Learns One-Layer Networks in WGANs
• Estimating Certain Integral Probability Metric (IPM) is as Hard as Estimating under the IPM
• The Local Elasticity of Neural Networks
• On the Minimax Optimality of Estimating the Wasserstein Metric
• Deep Learning for Individual Heterogeneity
• On nearly assumption-free tests of nominal confidence interval coverage for causal parameters estimated by machine learning
• Double Generative Adversarial Networks for Conditional Independence Testing
• Nonconvex sparse regularization for deep neural networks and its optimality
• Robust Density Estimation under Besov IPM Losses
• Posterior asymptotics in Wasserstein metrics on the real line
• Testing Directed Acyclic Graph via Structural, Supervised and Generative Adversarial Learning