Training Generative Adversarial Networks by Solving Ordinary Differential Equations

Abstract

The instability of Generative Adversarial Network (GAN) training has frequently been attributed to gradient descent. Consequently, recent methods have aimed to tailor the models and training procedures to stabilise the discrete updates. In contrast, we study the continuous-time dynamics induced by GAN training. Both theory and toy experiments suggest that these dynamics are in fact surprisingly stable. From this perspective, we hypothesise that instabilities in training GANs arise from the integration error in discretising the continuous dynamics. We experimentally verify that well-known ODE solvers (such as Runge-Kutta) can stabilise training - when combined with a regulariser that controls the integration error. Our approach represents a radical departure from previous methods which typically use adaptive optimisation and stabilisation techniques that constrain the functional space (e.g. Spectral Normalisation). Evaluation on CIFAR-10 and ImageNet shows that our method outperforms several strong baselines, demonstrating its efficacy.

References in corpus (14)

• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Spectral Normalization for Generative Adversarial Networks
• Which Training Methods for GANs do actually Converge?
• A Learned Representation For Artistic Style
• cGANs with Projection Discriminator
• Towards Principled Methods for Training Generative Adversarial Networks
• The Numerics of GANs
• The Deepfake Detection Challenge (DFDC) Preview Dataset
• The Mechanics of n-Player Differentiable Games
• Convergence of Learning Dynamics in Stackelberg Games
• Global Convergence to the Equilibrium of GANs using Variational Inequalities
• Adversarial Lipschitz Regularization
• A Solvable High-Dimensional Model of GAN

Cited by in corpus (6)

• A Systematic Survey of Regularization and Normalization in GANs
• Improving deep learning with prior knowledge and cognitive models: A survey on enhancing explainability, adversarial robustness and zero-shot learning
• Training Generative Adversarial Networks with Adaptive Composite Gradient
• Control of Overfitting with Physics
• Functional Space Analysis of Local GAN Convergence
• Complex Momentum for Optimization in Games