Forward Super-Resolution: How Can GANs Learn Hierarchical Generative Models for Real-World Distributions
arXiv:2106.02619
Abstract
Generative adversarial networks (GANs) are among the most successful models for learning high-complexity, real-world distributions. However, in theory, due to the highly non-convex, non-concave landscape of the minmax training objective, GAN remains one of the least understood deep learning models. In this work, we formally study how GANs can efficiently learn certain hierarchically generated distributions that are close to the distribution of real-life images. We prove that when a distribution has a structure that we refer to as Forward Super-Resolution, then simply training generative adversarial networks using stochastic gradient descent ascent (SGDA) can learn this distribution efficiently, both in sample and time complexities. We also provide empirical evidence that our assumption "forward super-resolution" is very natural in practice, and the underlying learning mechanisms that we study in this paper (to allow us efficiently train GAN via SGDA in theory) simulates the actual learning process of GANs on real-world problems.
v2 polishes writing
References in corpus (11)
- GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
- Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks
- Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networks
- Recovery Guarantees for One-hidden-layer Neural Networks
- Learning One-hidden-layer Neural Networks with Landscape Design
- Exact Recovery of Sparsely-Used Dictionaries
- Diverse Neural Network Learns True Target Functions
- Theoretical properties of the global optimizer of two layer neural network
- Recovery Guarantee of Non-negative Matrix Factorization via Alternating Updates
- Provable ICA with Unknown Gaussian Noise, and Implications for Gaussian Mixtures and Autoencoders
- Learning Over-Parametrized Two-Layer ReLU Neural Networks beyond NTK