On Self Modulation for Generative Adversarial Networks

Abstract

Training Generative Adversarial Networks (GANs) is notoriously challenging. We propose and study an architectural modification, self-modulation, which improves GAN performance across different data sets, architectures, losses, regularizers, and hyperparameter settings. Intuitively, self-modulation allows the intermediate feature maps of a generator to change as a function of the input noise vector. While reminiscent of other conditioning techniques, it requires no labeled data. In a large-scale empirical study we observe a relative decrease of $5\%-35\%$ in FID. Furthermore, all else being equal, adding this modification to the generator leads to improved performance in $124/144$ ($86\%$) of the studied settings. Self-modulation is a simple architectural change that requires no additional parameter tuning, which suggests that it can be applied readily to any GAN.

References in corpus (17)

• Adam: A Method for Stochastic Optimization
• Conditional Generative Adversarial Nets
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Spectral Normalization for Generative Adversarial Networks
• Instance Normalization: The Missing Ingredient for Fast Stylization
• Convolutional Sequence to Sequence Learning
• Self-Attention Generative Adversarial Networks
• Conditional Image Synthesis With Auxiliary Classifier GANs
• Conditional Image Generation with PixelCNN Decoders
• Progressive Growing of GANs for Improved Quality, Stability, and Variation
• LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop
• A Learned Representation For Artistic Style
• Modulating early visual processing by language
• A Note on the Inception Score
• Assessing Generative Models via Precision and Recall
• Is Generator Conditioning Causally Related to GAN Performance?
• Understanding the Effectiveness of Lipschitz-Continuity in Generative Adversarial Nets

Cited by in corpus (29)

• Semantic Image Synthesis with Spatially-Adaptive Normalization
• Parameter-Efficient Transfer Learning for NLP
• BlockGAN: Learning 3D Object-aware Scene Representations from Unlabelled Images
• Pretrained Transformers as Universal Computation Engines
• ViTGAN: Training GANs with Vision Transformers
• Unsupervised Object Segmentation by Redrawing
• Cross-Modal Contrastive Learning for Text-to-Image Generation
• A Large-Scale Study on Regularization and Normalization in GANs
• Feature Quantization Improves GAN Training
• Location-aware Adaptive Normalization: A Deep Learning Approach For Wildfire Danger Forecasting
• Adaptive Denoising via GainTuning
• Positional Normalization
• Self-Supervised GAN to Counter Forgetting
• Learning Spatial Pyramid Attentive Pooling in Image Synthesis and Image-to-Image Translation
• Self-Supervised GANs with Label Augmentation
• Self-Supervised GANs via Auxiliary Rotation Loss
• Self Sparse Generative Adversarial Networks
• Label-Noise Robust Multi-Domain Image-to-Image Translation
• Dual Contrastive Loss and Attention for GANs
• Improved Transformer for High-Resolution GANs
• Blur, Noise, and Compression Robust Generative Adversarial Networks
• Alleviation of Gradient Exploding in GANs: Fake Can Be Real
• Auto-Embedding Generative Adversarial Networks for High Resolution Image Synthesis
• Fine-grained Image-to-Image Transformation towards Visual Recognition
• CoPE: Conditional image generation using Polynomial Expansions
• Noise Robust Generative Adversarial Networks
• Unsupervised Image Transformation Learning via Generative Adversarial Networks
• Deep Automodulators
• Towards GANs' Approximation Ability