A Closer Look at the Optimization Landscapes of Generative Adversarial Networks

Abstract

Generative adversarial networks have been very successful in generative modeling, however they remain relatively challenging to train compared to standard deep neural networks. In this paper, we propose new visualization techniques for the optimization landscapes of GANs that enable us to study the game vector field resulting from the concatenation of the gradient of both players. Using these visualization techniques we try to bridge the gap between theory and practice by showing empirically that the training of GANs exhibits significant rotations around Local Stable Stationary Points (LSSP), similar to the one predicted by theory on toy examples. Moreover, we provide empirical evidence that GAN training converge to a stable stationary point which is a saddle point for the generator loss, not a minimum, while still achieving excellent performance.

References in corpus (16)

• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Spectral Normalization for Generative Adversarial Networks
• Which Training Methods for GANs do actually Converge?
• The Loss Surfaces of Multilayer Networks
• Identifying and attacking the saddle point problem in high-dimensional non-convex optimization
• Qualitatively characterizing neural network optimization problems
• Essentially No Barriers in Neural Network Energy Landscape
• The Numerics of GANs
• Empirical Analysis of the Hessian of Over-Parametrized Neural Networks
• The Mechanics of n-Player Differentiable Games
• Eigenvalues of the Hessian in Deep Learning: Singularity and Beyond
• The Unusual Effectiveness of Averaging in GAN Training
• On Finding Local Nash Equilibria (and Only Local Nash Equilibria) in Zero-Sum Games
• On the Limitations of First-Order Approximation in GAN Dynamics
• Local Saddle Point Optimization: A Curvature Exploitation Approach

Cited by in corpus (13)

• Convergence of Learning Dynamics in Stackelberg Games
• Adversarial Example Games
• Understanding Overparameterization in Generative Adversarial Networks
• Gradient Descent-Ascent Provably Converges to Strict Local Minmax Equilibria with a Finite Timescale Separation
• Smooth markets: A basic mechanism for organizing gradient-based learners
• On Characterizing GAN Convergence Through Proximal Duality Gap
• A Tight and Unified Analysis of Gradient-Based Methods for a Whole Spectrum of Games
• Taming GANs with Lookahead-Minmax
• Fair Normalizing Flows
• A Differential Geometry Perspective on Orthogonal Recurrent Models
• A Limited-Capacity Minimax Theorem for Non-Convex Games or: How I Learned to Stop Worrying about Mixed-Nash and Love Neural Nets
• Stability of Gradient Learning Dynamics in Continuous Games: Vector Action Spaces
• Complex Momentum for Optimization in Games