Making Method of Moments Great Again? -- How can GANs learn distributions
arXiv:2003.04033
Abstract
Generative Adversarial Networks (GANs) are widely used models to learn complex real-world distributions. In GANs, the training of the generator usually stops when the discriminator can no longer distinguish the generator's output from the set of training examples. A central question of GANs is that when the training stops, whether the generated distribution is actually close to the target distribution, and how the training process reaches to such configurations efficiently? In this paper, we established a theoretical results towards understanding this generator-discriminator training process. We empirically observe that during the earlier stage of the GANs training, the discriminator is trying to force the generator to match the low degree moments between the generator's output and the target distribution. Moreover, only by matching these empirical moments over polynomially many training examples, we prove that the generator can already learn notable class of distributions, including those that can be generated by two-layer neural networks.
arXiv comment50+ pages
References in corpus (35)
- Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
- GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
- Spectral Normalization for Generative Adversarial Networks
- Self-Attention Generative Adversarial Networks
- Conditional Image Synthesis With Auxiliary Classifier GANs
- Neural Tangent Kernel: Convergence and Generalization in Neural Networks
- A Convergence Theory for Deep Learning via Over-Parameterization
- MMD GAN: Towards Deeper Understanding of Moment Matching Network
- On Exact Computation with an Infinitely Wide Neural Net
- Gradient Descent Provably Optimizes Over-parameterized Neural Networks
- Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Data
- Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks
- Generalization and Equilibrium in Generative Adversarial Nets (GANs)
- Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivity
- On Gradient Descent Ascent for Nonconvex-Concave Minimax Problems
- The Numerics of GANs
- Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networks
- Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation
- Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learning
- No bad local minima: Data independent training error guarantees for multilayer neural networks
- A Unified Analysis of Extra-gradient and Optimistic Gradient Methods for Saddle Point Problems: Proximal Point Approach
- Recovery Guarantees for One-hidden-layer Neural Networks
- Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networks
- On the Convergence Rate of Training Recurrent Neural Networks
- A Priori Estimates of the Population Risk for Two-layer Neural Networks
- What Can ResNet Learn Efficiently, Going Beyond Kernels?
- An Analytical Formula of Population Gradient for two-layered ReLU network and its Applications in Convergence and Critical Point Analysis
- Learning One-hidden-layer ReLU Networks via Gradient Descent
- Understanding GANs: the LQG Setting
- Approximability of Discriminators Implies Diversity in GANs
- Theoretical properties of the global optimizer of two layer neural network
- Recovery Guarantee of Non-negative Matrix Factorization via Alternating Updates
- Provable Alternating Gradient Descent for Non-negative Matrix Factorization with Strong Correlations
- SGD Learns One-Layer Networks in WGANs
- Learning Over-Parametrized Two-Layer ReLU Neural Networks beyond NTK