The Numerics of GANs

Abstract

In this paper, we analyze the numerics of common algorithms for training Generative Adversarial Networks (GANs). Using the formalism of smooth two-player games we analyze the associated gradient vector field of GAN training objectives. Our findings suggest that the convergence of current algorithms suffers due to two factors: i) presence of eigenvalues of the Jacobian of the gradient vector field with zero real-part, and ii) eigenvalues with big imaginary part. Using these findings, we design a new algorithm that overcomes some of these limitations and has better convergence properties. Experimentally, we demonstrate its superiority on training common GAN architectures and show convergence on GAN architectures that are known to be notoriously hard to train.

References in corpus (4)

• Improved Training of Wasserstein GANs
• Adversarial Discriminative Domain Adaptation
• Amortised MAP Inference for Image Super-resolution
• Connecting Generative Adversarial Networks and Actor-Critic Methods

Cited by in corpus (107)

• Generative Adversarial Networks: An Overview
• A Variational Inequality Perspective on Generative Adversarial Networks
• Game-Theoretic Multiagent Reinforcement Learning
• The Mechanics of n-Player Differentiable Games
• A Dual-Dimer Method for Training Physics-Constrained Neural Networks with Minimax Architecture
• The Unusual Effectiveness of Averaging in GAN Training
• On Finding Local Nash Equilibria (and Only Local Nash Equilibria) in Zero-Sum Games
• Stabilizing Generative Adversarial Networks: A Survey
• Maximum Entropy Generators for Energy-Based Models
• Reducing Noise in GAN Training with Variance Reduced Extragradient
• LOGAN: Latent Optimisation for Generative Adversarial Networks
• Out of Distribution Generalization in Machine Learning
• Competitive Gradient Descent
• Convergence of Learning Dynamics in Stackelberg Games
• Lipschitz Generative Adversarial Nets
• Last-iterate convergence rates for min-max optimization
• A Closer Look at the Optimization Landscapes of Generative Adversarial Networks
• High-resolution Deep Convolutional Generative Adversarial Networks
• GANs May Have No Nash Equilibria
• Advances in Variational Inference
• Global Convergence to the Equilibrium of GANs using Variational Inequalities
• Painless Stochastic Gradient: Interpolation, Line-Search, and Convergence Rates
• On Catastrophic Forgetting and Mode Collapse in Generative Adversarial Networks
• Cali-Sketch: Stroke Calibration and Completion for High-Quality Face Image Generation from Human-Like Sketches
• Encoding Invariances in Deep Generative Models
• Towards Better Understanding of Adaptive Gradient Algorithms in Generative Adversarial Nets
• Accelerating Smooth Games by Manipulating Spectral Shapes
• Adversarial Variational Optimization of Non-Differentiable Simulators
• On Solving Minimax Optimization Locally: A Follow-the-Ridge Approach
• Saddle Point Optimization with Approximate Minimization Oracle and its Application to Robust Berthing Control
• From Poincaré Recurrence to Convergence in Imperfect Information Games: Finding Equilibrium via Regularization
• Improved Algorithms for Convex-Concave Minimax Optimization
• Negative Momentum for Improved Game Dynamics
• Implicit competitive regularization in GANs
• Manifold regularization with GANs for semi-supervised learning
• Improving the Speed and Quality of GAN by Adversarial Training
• Training Generative Adversarial Networks by Solving Ordinary Differential Equations
• SGD Learns One-Layer Networks in WGANs
• Understanding Overparameterization in Generative Adversarial Networks
• A Unified Analysis of First-Order Methods for Smooth Games via Integral Quadratic Constraints
• Newton-type Methods for Minimax Optimization
• Interaction-aware Multi-agent Tracking and Probabilistic Behavior Prediction via Adversarial Learning
• Regularizing Generative Adversarial Networks under Limited Data
• Investigating Under and Overfitting in Wasserstein Generative Adversarial Networks
• On the Suboptimality of Negative Momentum for Minimax Optimization
• Stable Opponent Shaping in Differentiable Games
• On the Impossibility of Global Convergence in Multi-Loss Optimization
• An Improved Self-supervised GAN via Adversarial Training
• Interaction Matters: A Note on Non-asymptotic Local Convergence of Generative Adversarial Networks
• Understanding the Effectiveness of Lipschitz-Continuity in Generative Adversarial Nets
• Non-saturating GAN training as divergence minimization
• Smooth markets: A basic mechanism for organizing gradient-based learners
• Understanding and Stabilizing GANs' Training Dynamics with Control Theory
• Gradient Descent-Ascent Provably Converges to Strict Local Minmax Equilibria with a Finite Timescale Separation
• Solving Structured Hierarchical Games Using Differential Backward Induction
• Saddle Point Optimization with Approximate Minimization Oracle
• Optimality and Stability in Non-Convex Smooth Games
• Near-optimal Local Convergence of Alternating Gradient Descent-Ascent for Minimax Optimization
• Train simultaneously, generalize better: Stability of gradient-based minimax learners
• Designing GANs: A Likelihood Ratio Approach
• Towards Unsupervised Learning of Generative Models for 3D Controllable Image Synthesis
• Properties of f-divergences and f-GAN training
• Taming GANs with Lookahead-Minmax
• Nonconvex-Nonconcave Min-Max Optimization with a Small Maximization Domain
• Implicit Gradient Regularization
• Generative Ratio Matching Networks
• Adaptive Weighted Discriminator for Training Generative Adversarial Networks
• Extragradient with player sampling for faster Nash equilibrium finding
• Forward Super-Resolution: How Can GANs Learn Hierarchical Generative Models for Real-World Distributions
• Neural Lyapunov Redesign
• DO-GAN: A Double Oracle Framework for Generative Adversarial Networks
• not-so-BigGAN: Generating High-Fidelity Images on Small Compute with Wavelet-based Super-Resolution
• Minimax Problems with Coupled Linear Constraints: Computational Complexity, Duality and Solution Methods
• Consensus Multiplicative Weights Update: Learning to Learn using Projector-based Game Signatures
• Generative Adversarial Networks and Adversarial Autoencoders: Tutorial and Survey
• Don't Generate Me: Training Differentially Private Generative Models with Sinkhorn Divergence
• FIS-GAN: GAN with Flow-based Importance Sampling
• Stochastic Gradient Descent-Ascent and Consensus Optimization for Smooth Games: Convergence Analysis under Expected Co-coercivity
• Omni-GAN: On the Secrets of cGANs and Beyond
• A Differential Geometry Perspective on Orthogonal Recurrent Models
• Competitive Mirror Descent
• A Differential Game Theoretic Neural Optimizer for Training Residual Networks
• Primal-Dual Sequential Subspace Optimization for Saddle-point Problems
• SyMetric: Measuring the Quality of Learnt Hamiltonian Dynamics Inferred from Vision
• Noise Homogenization via Multi-Channel Wavelet Filtering for High-Fidelity Sample Generation in GANs
• Online Kernel based Generative Adversarial Networks
• Game of GANs: Game-Theoretical Models for Generative Adversarial Networks
• Making Method of Moments Great Again? -- How can GANs learn distributions
• Minimax Optimization with Smooth Algorithmic Adversaries
• Mode Penalty Generative Adversarial Network with adapted Auto-encoder
• MTAdam: Automatic Balancing of Multiple Training Loss Terms
• Complex Momentum for Optimization in Games
• Deep Direct Likelihood Knockoffs
• Implementation of Optical Deep Neural Networks using the Fabry-Perot Interferometer
• LRS-DAG: Low Resource Supervised Domain Adaptation with Generalization Across Domains
• High Fidelity Semantic Shape Completion for Point Clouds using Latent Optimization
• When Relation Networks meet GANs: Relation GANs with Triplet Loss
• Scrutinizing and De-Biasing Intuitive Physics with Neural Stethoscopes
• FusedProp: Towards Efficient Training of Generative Adversarial Networks
• Towards GANs' Approximation Ability
• Polymatrix Competitive Gradient Descent
• The Geometric Occam's Razor Implicit in Deep Learning
• Stochastic Projective Splitting: Solving Saddle-Point Problems with Multiple Regularizers
• The Benefits of Pairwise Discriminators for Adversarial Training
• Reproducibility Challenge NeurIPS 2019 Report on "Competitive Gradient Descent"
• Coulomb Autoencoders
• Strategic Prediction with Latent Aggregative Games