A Systematic Survey of Regularization and Normalization in GANs

Abstract

Generative Adversarial Networks (GANs) have been widely applied in different scenarios thanks to the development of deep neural networks. The original GAN was proposed based on the non-parametric assumption of the infinite capacity of networks. However, it is still unknown whether GANs can fit the target distribution without any prior information. Due to the overconfident assumption, many issues remain unaddressed in GANs' training, such as non-convergence, mode collapses, gradient vanishing. Regularization and normalization are common methods of introducing prior information to stabilize training and improve discrimination. Although a handful number of regularization and normalization methods have been proposed for GANs, to the best of our knowledge, there exists no comprehensive survey that primarily focuses on objectives and development of these methods, apart from some in-comprehensive and limited scope studies. In this work, we conduct a comprehensive survey on the regularization and normalization techniques from different perspectives of GANs training. First, we systematically describe different perspectives of GANs training and thus obtain the different objectives of regularization and normalization. Based on these objectives, we propose a new taxonomy. Furthermore, we compare the performance of the mainstream methods on different datasets and investigate the applications of regularization and normalization techniques that have been frequently employed in state-of-the-art GANs. Finally, we highlight potential future directions of research in this domain. Code and studies related to the regularization and normalization of GANs in this work is summarized on https://github.com/iceli1007/GANs-Regularization-Review.

arXiv commentAccepted to Acm Computing Survey

References in corpus (74)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Representation Learning with Contrastive Predictive Coding
• mixup: Beyond Empirical Risk Minimization
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Spectral Normalization for Generative Adversarial Networks
• Instance Normalization: The Missing Ingredient for Fast Stylization
• Improved Regularization of Convolutional Neural Networks with Cutout
• Self-Attention Generative Adversarial Networks
• FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence
• Diffusion Models Beat GANs on Image Synthesis
• Improved Baselines with Momentum Contrastive Learning
• Generative Adversarial Network in Medical Imaging: A Review
• Learning deep representations by mutual information estimation and maximization
• Unsupervised Data Augmentation for Consistency Training
• Progressive Growing of GANs for Improved Quality, Stability, and Variation
• Unsupervised Representation Learning by Predicting Image Rotations
• Training Generative Adversarial Networks with Limited Data
• Which Training Methods for GANs do actually Converge?
• The relativistic discriminator: a key element missing from standard GAN
• cGANs with Projection Discriminator
• Perceptual Adversarial Networks for Image-to-Image Transformation
• Towards Principled Methods for Training Generative Adversarial Networks
• Neural Photo Editing with Introspective Adversarial Networks
• Regularization for Deep Learning: A Taxonomy
• DRIT++: Diverse Image-to-Image Translation via Disentangled Representations
• A Variational Inequality Perspective on Generative Adversarial Networks
• Large Scale Image Completion via Co-Modulated Generative Adversarial Networks
• On the regularization of Wasserstein GANs
• A survey on text generation using generative adversarial networks
• Generative Adversarial Networks for Spatio-temporal Data: A Survey
• Diversity-Sensitive Conditional Generative Adversarial Networks
• Image Augmentations for GAN Training
• Many Paths to Equilibrium: GANs Do Not Need to Decrease a Divergence At Every Step
• Differentiable Augmentation for Data-Efficient GAN Training
• Patch-Based Image Inpainting with Generative Adversarial Networks
• Improving Generalization and Stability of Generative Adversarial Networks
• Bias and Generalization in Deep Generative Models: An Empirical Study
• On the Effects of Batch and Weight Normalization in Generative Adversarial Networks
• Diffusion-GAN: Training GANs with Diffusion
• CSCNet: Contextual Semantic Consistency Network for Trajectory Prediction in Crowded Spaces
• Improving the Improved Training of Wasserstein GANs: A Consistency Term and Its Dual Effect
• Is Generator Conditioning Causally Related to GAN Performance?
• Consistency Regularization for Generative Adversarial Networks
• Probabilistic Feature Selection and Classification Vector Machine
• Competitive Gradient Descent
• On the Limitations of First-Order Approximation in GAN Dynamics
• Lipschitz Generative Adversarial Nets
• Generative Adversarial Networks for Image Super-Resolution: A Survey
• Data-Efficient Instance Generation from Instance Discrimination
• A Large-Scale Study on Regularization and Normalization in GANs
• Simple yet Effective Way for Improving the Performance of GAN
• Deceive D: Adaptive Pseudo Augmentation for GAN Training with Limited Data
• GAN-QP: A Novel GAN Framework without Gradient Vanishing and Lipschitz Constraint
• Regularization Methods for Generative Adversarial Networks: An Overview of Recent Studies
• A Comprehensive Survey on Data-Efficient GANs in Image Generation
• On Solving Minimax Optimization Locally: A Follow-the-Ridge Approach
• How Well Do WGANs Estimate the Wasserstein Metric?
• Real or Not Real, that is the Question
• Wasserstein GANs Work Because They Fail (to Approximate the Wasserstein Distance)
• SSD-GAN: Measuring the Realness in the Spatial and Spectral Domains
• Consistency-based Semi-supervised Active Learning: Towards Minimizing Labeling Cost
• Training Generative Adversarial Networks by Solving Ordinary Differential Equations
• Regularizing Generative Adversarial Networks under Limited Data
• Stabilizing Adversarial Nets With Prediction Methods
• Training Generative Adversarial Networks Via Turing Test
• Improving GAN Training with Probability Ratio Clipping and Sample Reweighting
• Towards Efficient and Unbiased Implementation of Lipschitz Continuity in GANs
• DeshuffleGAN: A Self-Supervised GAN to Improve Structure Learning
• A Wasserstein GAN model with the total variational regularization
• View Vertically: A Hierarchical Network for Trajectory Prediction via Fourier Spectrums
• Augmented Cyclic Consistency Regularization for Unpaired Image-to-Image Translation
• FakeCLR: Exploring Contrastive Learning for Solving Latent Discontinuity in Data-Efficient GANs
• InfoMax-GAN: Improved Adversarial Image Generation via Information Maximization and Contrastive Learning

Cited by in corpus (1)

• Fault Detection in New Wind Turbines with Limited Data by Generative Transfer Learning