WGAN with an Infinitely Wide Generator Has No Spurious Stationary Points
arXiv:2102.07541
Abstract
Generative adversarial networks (GAN) are a widely used class of deep generative models, but their minimax training dynamics are not understood very well. In this work, we show that GANs with a 2-layer infinite-width generator and a 2-layer finite-width discriminator trained with stochastic gradient ascent-descent have no spurious stationary points. We then show that when the width of the generator is finite but wide, there are no spurious stationary points within a ball whose radius becomes arbitrarily large (to cover the entire parameter space) as the width goes to infinity.
arXiv commentPublished at ICML 2021
References in corpus (17)
- Neural Tangent Kernel: Convergence and Generalization in Neural Networks
- NIPS 2016 Tutorial: Generative Adversarial Networks
- Which Training Methods for GANs do actually Converge?
- Deep Neural Networks as Gaussian Processes
- A Mean Field View of the Landscape of Two-Layers Neural Networks
- Gradient Descent Provably Optimizes Over-parameterized Neural Networks
- Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Data
- How to Escape Saddle Points Efficiently
- The Global Landscape of Neural Networks: An Overview
- On the Convergence and Robustness of Training GANs with Regularized Optimal Transport
- On the loss landscape of a class of deep neural networks with no bad local valleys
- Understanding the Loss Surface of Neural Networks for Binary Classification
- Adding One Neuron Can Eliminate All Bad Local Minima
- Explaining Landscape Connectivity of Low-cost Solutions for Multilayer Nets
- On the Benefit of Width for Neural Networks: Disappearance of Bad Basins
- Towards a Better Global Loss Landscape of GANs
- When Does Non-Orthogonal Tensor Decomposition Have No Spurious Local Minima?