Which Training Methods for GANs do actually Converge?

Abstract

Recent work has shown local convergence of GAN training for absolutely continuous data and generator distributions. In this paper, we show that the requirement of absolute continuity is necessary: we describe a simple yet prototypical counterexample showing that in the more realistic case of distributions that are not absolutely continuous, unregularized GAN training is not always convergent. Furthermore, we discuss regularization strategies that were recently proposed to stabilize GAN training. Our analysis shows that GAN training with instance noise or zero-centered gradient penalties converges. On the other hand, we show that Wasserstein-GANs and WGAN-GP with a finite number of discriminator updates per generator update do not always converge to the equilibrium point. We discuss these results, leading us to a new explanation for the stability problems of GAN training. Based on our analysis, we extend our convergence results to more general GANs and prove local convergence for simplified gradient penalties even if the generator and data distribution lie on lower dimensional manifolds. We find these penalties to work well in practice and use them to learn high-resolution generative image models for a variety of datasets with little hyperparameter tuning.

conference

Cited by in corpus (240)

• Training Generative Adversarial Networks with Limited Data
• AMP: Adversarial Motion Priors for Stylized Physics-Based Character Control
• GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis
• Improved Precision and Recall Metric for Assessing Generative Models
• Deep Learning Approaches for Data Augmentation in Medical Imaging: A Review
• TransGAN: Two Pure Transformers Can Make One Strong GAN, and That Can Scale Up
• Semantic Image Synthesis with Spatially-Adaptive Normalization
• Adaptive Diffusion Priors for Accelerated MRI Reconstruction
• StyleNeRF: A Style-based 3D-Aware Generator for High-resolution Image Synthesis
• How to GAN LHC Events
• A Variational Inequality Perspective on Generative Adversarial Networks
• Swapping Autoencoder for Deep Image Manipulation
• Game-Theoretic Multiagent Reinforcement Learning
• Variational Discriminator Bottleneck: Improving Imitation Learning, Inverse RL, and GANs by Constraining Information Flow
• Contrastive Learning for Unpaired Image-to-Image Translation
• Train Sparsely, Generate Densely: Memory-efficient Unsupervised Training of High-resolution Temporal GAN
• Deep unsupervised learning of turbulence for inflow generation at various Reynolds numbers
• Review: Deep Learning in Electron Microscopy
• From Variational to Deterministic Autoencoders
• The Unusual Effectiveness of Averaging in GAN Training
• Differentiable Augmentation for Data-Efficient GAN Training
• StarGAN v2: Diverse Image Synthesis for Multiple Domains
• ContraGAN: Contrastive Learning for Conditional Image Generation
• Joint Discriminative and Generative Learning for Person Re-identification
• Few-Shot Unsupervised Image-to-Image Translation
• ViTGAN: Training GANs with Vision Transformers
• CIPS-3D: A 3D-Aware Generator of GANs Based on Conditionally-Independent Pixel Synthesis
• Generative Steganography Network
• Stabilizing Generative Adversarial Networks: A Survey
• SofGAN: A Portrait Image Generator with Dynamic Styling
• A Systematic Survey of Regularization and Normalization in GANs
• On Leveraging Pretrained GANs for Generation with Limited Data
• LOGAN: Latent Optimisation for Generative Adversarial Networks
• On gradient regularizers for MMD GANs
• StyleCariGAN: Caricature Generation via StyleGAN Feature Map Modulation
• Machine Learning for Physics-Informed Generation of Dispersed Multiphase Flow Using Generative Adversarial Networks
• medigan: a Python library of pretrained generative models for medical image synthesis
• DataLens: Scalable Privacy Preserving Training via Gradient Compression and Aggregation
• COutfitGAN: Learning to Synthesize Compatible Outfits Supervised by Silhouette Masks and Fashion Styles
• Generating Multi-Categorical Samples with Generative Adversarial Networks
• Convergence of Learning Dynamics in Stackelberg Games
• Lipschitz Generative Adversarial Nets
• Lower Dimensional Kernels for Video Discriminators
• Last-iterate convergence rates for min-max optimization
• Active Domain Randomization
• Generative Adversarial Networks for Image and Video Synthesis: Algorithms and Applications
• Synthetic flow-based cryptomining attack generation through Generative Adversarial Networks
• A Large-Scale Study on Regularization and Normalization in GANs
• A Closer Look at the Optimization Landscapes of Generative Adversarial Networks
• PConv: Simple yet Effective Convolutional Layer for Generative Adversarial Network
• How Well Generative Adversarial Networks Learn Distributions
• Focal Frequency Loss for Image Reconstruction and Synthesis
• Feature Quantization Improves GAN Training
• Deceive D: Adaptive Pseudo Augmentation for GAN Training with Limited Data
• Arbitrary Talking Face Generation via Attentional Audio-Visual Coherence Learning
• Explore Aggressively, Update Conservatively: Stochastic Extragradient Methods with Variable Stepsize Scaling
• High-resolution Deep Convolutional Generative Adversarial Networks
• Global Convergence to the Equilibrium of GANs using Variational Inequalities
• On Catastrophic Forgetting and Mode Collapse in Generative Adversarial Networks
• Can AI be enabled to dynamical downscaling? A Latent Diffusion Model to mimic km-scale COSMO5.0\_CLM9 simulations
• MSG-GAN: Multi-Scale Gradients for Generative Adversarial Networks
• Mode Collapse and Regularity of Optimal Transportation Maps
• Latent Denoising Diffusion GAN: Faster sampling, Higher image quality
• Fast and Scalable Earth Texture Synthesis using Spatially Assembled Generative Adversarial Neural Networks
• Image Outpainting and Harmonization using Generative Adversarial Networks
• Accelerating Smooth Games by Manipulating Spectral Shapes
• InfinityGAN: Towards Infinite-Pixel Image Synthesis
• Fast and Provable ADMM for Learning with Generative Priors
• Smoothness and Stability in GANs
• GANeRF: Leveraging Discriminators to Optimize Neural Radiance Fields
• Encoding Invariances in Deep Generative Models
• COVID-19 CT Image Synthesis with a Conditional Generative Adversarial Network
• Unsupervised Discovery of Object Radiance Fields
• Towards a Better Understanding and Regularization of GAN Training Dynamics
• A Generative Model for Texture Synthesis based on Optimal Transport between Feature Distributions
• CariGAN: Caricature Generation through Weakly Paired Adversarial Learning
• InMoDeGAN: Interpretable Motion Decomposition Generative Adversarial Network for Video Generation
• Lipschitzness Is All You Need To Tame Off-policy Generative Adversarial Imitation Learning
• A Spectral Regularizer for Unsupervised Disentanglement
• Interactive Sketch & Fill: Multiclass Sketch-to-Image Translation
• Human Motion Prediction via Spatio-Temporal Inpainting
• Compressing PDF sets using generative adversarial networks
• Negative Momentum for Improved Game Dynamics
• RGBD-GAN: Unsupervised 3D Representation Learning From Natural Image Datasets via RGBD Image Synthesis
• Generative Adversarial Networks (GANs): What it can generate and What it cannot?
• CGOF++: Controllable 3D Face Synthesis with Conditional Generative Occupancy Fields
• An Imitation from Observation Approach to Transfer Learning with Dynamics Mismatch
• Self-Diagnosing GAN: Diagnosing Underrepresented Samples in Generative Adversarial Networks
• Efficient Generation of Structured Objects with Constrained Adversarial Networks
• From Inference to Generation: End-to-end Fully Self-supervised Generation of Human Face from Speech
• Content and Colour Distillation for Learning Image Translations with the Spatial Profile Loss
• Scalable Unbalanced Optimal Transport using Generative Adversarial Networks
• Manifold regularization with GANs for semi-supervised learning
• Generative Models as Distributions of Functions
• Training Generative Adversarial Networks by Solving Ordinary Differential Equations
• SGD Learns One-Layer Networks in WGANs
• Text-only domain adaptation for end-to-end ASR using integrated text-to-mel-spectrogram generator
• Training Deep Energy-Based Models with f-Divergence Minimization
• Regularizing Generative Adversarial Networks under Limited Data
• GAN-powered Deep Distributional Reinforcement Learning for Resource Management in Network Slicing
• LatteGAN: Visually Guided Language Attention for Multi-Turn Text-Conditioned Image Manipulation
• GraN-GAN: Piecewise Gradient Normalization for Generative Adversarial Networks
• COCO-GAN: Generation by Parts via Conditional Coordinating
• The Spatially-Correlative Loss for Various Image Translation Tasks
• Image-to-image Translation via Hierarchical Style Disentanglement
• Task-Aware Feature Generation for Zero-Shot Compositional Learning
• An Improved Self-supervised GAN via Adversarial Training
• How does Lipschitz Regularization Influence GAN Training?
• COCO-FUNIT: Few-Shot Unsupervised Image Translation with a Content Conditioned Style Encoder
• MRI Banding Removal via Adversarial Training
• Understanding the Effectiveness of Lipschitz-Continuity in Generative Adversarial Nets
• Self-Supervised GAN to Counter Forgetting
• Group Equivariant Generative Adversarial Networks
• Non-saturating GAN training as divergence minimization
• HyperInverter: Improving StyleGAN Inversion via Hypernetwork
• Dynamics of Fourier Modes in Torus Generative Adversarial Networks
• Training Federated GANs with Theoretical Guarantees: A Universal Aggregation Approach
• Gradient Descent-Ascent Provably Converges to Strict Local Minmax Equilibria with a Finite Timescale Separation
• AniGAN: Style-Guided Generative Adversarial Networks for Unsupervised Anime Face Generation
• Neural Architecture Search for Generative Adversarial Networks: A Comprehensive Review and Critical Analysis
• Data-Efficient GAN Training Beyond (Just) Augmentations: A Lottery Ticket Perspective
• Self Sparse Generative Adversarial Networks
• Understanding and Stabilizing GANs' Training Dynamics with Control Theory
• Label-Noise Robust Generative Adversarial Networks
• Self-Supervised GANs via Auxiliary Rotation Loss
• How Does GAN-based Semi-supervised Learning Work?
• Follow-the-Regularized-Leader Routes to Chaos in Routing Games
• PE-GAN: Prior Embedding GAN for PXD images at Belle II
• Site-Specific Ground Motion Generative Model for Crustal Earthquakes in Japan Based on Generative Adversarial Networks
• Tdcgan: Temporal Dilated Convolutional Generative Adversarial Network for End-to-end Speech Enhancement
• Discriminator Feature-based Inference by Recycling the Discriminator of GANs
• PAGAN: Portfolio Analysis with Generative Adversarial Networks
• Train simultaneously, generalize better: Stability of gradient-based minimax learners
• Benchmarks, Algorithms, and Metrics for Hierarchical Disentanglement
• Robust Imitation Learning from Noisy Demonstrations
• GAN Cocktail: mixing GANs without dataset access
• High Resolution Face Editing with Masked GAN Latent Code Optimization
• Meta Learning for Few-Shot One-class Classification
• An Acceleration Framework for High Resolution Image Synthesis
• First Order Generative Adversarial Networks
• Poincaré Recurrence, Cycles and Spurious Equilibria in Gradient-Descent-Ascent for Non-Convex Non-Concave Zero-Sum Games
• ComicGAN: Text-to-Comic Generative Adversarial Network
• Properties of f-divergences and f-GAN training
• Towards Unsupervised Learning of Generative Models for 3D Controllable Image Synthesis
• Generating artificial digital image correlation data using physics-guided adversarial networks
• A Pragmatic Look at Deep Imitation Learning
• Disentangling representations of retinal images with generative models
• Towards Principled Disentanglement for Domain Generalization
• MatchingGAN: Matching-based Few-shot Image Generation
• SPG-VTON: Semantic Prediction Guidance for Multi-pose Virtual Try-on
• Taming GANs with Lookahead-Minmax
• Unsupervised Eyeglasses Removal in the Wild
• DeltaGAN: Towards Diverse Few-shot Image Generation with Sample-Specific Delta
• CAM-GAN: Continual Adaptation Modules for Generative Adversarial Networks
• A Mathematical Framework for Learning Probability Distributions
• Where is the Fake? Patch-Wise Supervised GANs for Texture Inpainting
• Multiqubit state learning with entangling quantum generative adversarial networks
• MPG: A Multi-ingredient Pizza Image Generator with Conditional StyleGANs
• Adaptive Weighted Discriminator for Training Generative Adversarial Networks
• Efficient Learning of Generative Models via Finite-Difference Score Matching
• From Shadow Segmentation to Shadow Removal
• BalaGAN: Image Translation Between Imbalanced Domains via Cross-Modal Transfer
• Cross-Sensor Adversarial Domain Adaptation of Landsat-8 and Proba-V images for Cloud Detection
• Improved Transformer for High-Resolution GANs
• GAN Memory with No Forgetting
• Reciprocal Adversarial Learning via Characteristic Functions
• Hidden Convexity of Wasserstein GANs: Interpretable Generative Models with Closed-Form Solutions
• Learning Neural Light Transport
• Class-Distinct and Class-Mutual Image Generation with GANs
• Image-to-Image Translation with Low Resolution Conditioning
• Face Reconstruction with Variational Autoencoder and Face Masks
• Don't Generate Me: Training Differentially Private Generative Models with Sinkhorn Divergence
• not-so-BigGAN: Generating High-Fidelity Images on Small Compute with Wavelet-based Super-Resolution
• Accelerated WGAN update strategy with loss change rate balancing
• Lessons Learned from the Training of GANs on Artificial Datasets
• The use of Generative Adversarial Networks to characterise new physics in multi-lepton final states at the LHC
• Instance Selection for GANs
• Diverse Image Generation via Self-Conditioned GANs
• Double Descent and Other Interpolation Phenomena in GANs
• Perceptual Image Super-Resolution with Progressive Adversarial Network
• Blur, Noise, and Compression Robust Generative Adversarial Networks
• A Benchmarking Initiative for Audio-Domain Music Generation Using the Freesound Loop Dataset
• A Novel Framework for Selection of GANs for an Application
• Alleviation of Gradient Exploding in GANs: Fake Can Be Real
• Generative Adversarial Networks and Adversarial Autoencoders: Tutorial and Survey
• Gradient penalty from a maximum margin perspective
• Optimal 1-Wasserstein Distance for WGANs
• Curriculum Learning for Deep Generative Models with Clustering
• Omni-GAN: On the Secrets of cGANs and Beyond
• Efficient Heuristic Generation for Robot Path Planning with Recurrent Generative Model
• FaceDet3D: Facial Expressions with 3D Geometric Detail Prediction
• SLGAN: Style- and Latent-guided Generative Adversarial Network for Desirable Makeup Transfer and Removal
• Lipschitz Constrained GANs via Boundedness and Continuity
• Unsupervised Object Learning via Common Fate
• Adaptive Divergence for Rapid Adversarial Optimization
• SyMetric: Measuring the Quality of Learnt Hamiltonian Dynamics Inferred from Vision
• Amortised Learning by Wake-Sleep
• Semisupervised Adversarial Neural Networks for Cyber Security Transfer Learning
• Minimax Optimization with Smooth Algorithmic Adversaries
• Reliable Estimation of KL Divergence using a Discriminator in Reproducing Kernel Hilbert Space
• Kernel Mean Matching for Content Addressability of GANs
• Conditioning Trick for Training Stable GANs
• Reparameterized Variational Divergence Minimization for Stable Imitation
• Semi-Anchored Multi-Step Gradient Descent Ascent Method for Structured Nonconvex-Nonconcave Composite Minimax Problems
• Exploiting Spatial Dimensions of Latent in GAN for Real-time Image Editing
• Domain Partitioning Network
• GANSeg: Learning to Segment by Unsupervised Hierarchical Image Generation
• LatentKeypointGAN: Controlling Images via Latent Keypoints
• Stochastic Projective Splitting: Solving Saddle-Point Problems with Multiple Regularizers
• Generative Adversarial Learning via Kernel Density Discrimination
• Analysis of Discriminator in RKHS Function Space for Kullback-Leibler Divergence Estimation
• When Relation Networks meet GANs: Relation GANs with Triplet Loss
• Complex Momentum for Optimization in Games
• Understanding Entropic Regularization in GANs
• NeurInt : Learning to Interpolate through Neural ODEs
• The Benefits of Pairwise Discriminators for Adversarial Training
• WGAN with an Infinitely Wide Generator Has No Spurious Stationary Points
• Enhanced 3DMM Attribute Control via Synthetic Dataset Creation Pipeline
• FusedProp: Towards Efficient Training of Generative Adversarial Networks
• Towards GANs' Approximation Ability
• The Geometric Occam's Razor Implicit in Deep Learning
• Local Stability and Performance of Simple Gradient Penalty mu-Wasserstein GAN
• Topics to Avoid: Demoting Latent Confounds in Text Classification
• Training End-to-end Single Image Generators without GANs
• Dual Projection Generative Adversarial Networks for Conditional Image Generation
• Toward a Generalization Metric for Deep Generative Models
• Noise Robust Generative Adversarial Networks
• Label Geometry Aware Discriminator for Conditional Generative Networks
• Convolutional generative adversarial imputation networks for spatio-temporal missing data in storm surge simulations
• MixSyn: Learning Composition and Style for Multi-Source Image Synthesis
• Toward Zero-Shot Unsupervised Image-to-Image Translation
• Generation and Simulation of Yeast Microscopy Imagery with Deep Learning
• Capturing Evolution Genes for Time Series Data
• A Kernel Test for Causal Association via Noise Contrastive Backdoor Adjustment
• Translate the Facial Regions You Like Using Region-Wise Normalization
• Creation of Synthetic Networked PMU Data: A Generative Adversarial Network Approach
• Convergence of GANs Training: A Game and Stochastic Control Methodology
• Synthesizing Photorealistic Images with Deep Generative Learning
• Generative Adversarial Network: Some Analytical Perspectives
• Efficient training for future video generation based on hierarchical disentangled representation of latent variables