Data-Efficient GAN Training Beyond (Just) Augmentations: A Lottery Ticket Perspective

Abstract

Training generative adversarial networks (GANs) with limited real image data generally results in deteriorated performance and collapsed models. To conquer this challenge, we are inspired by the latest observation, that one can discover independently trainable and highly sparse subnetworks (a.k.a., lottery tickets) from GANs. Treating this as an inductive prior, we suggest a brand-new angle towards data-efficient GAN training: by first identifying the lottery ticket from the original GAN using the small training set of real images; and then focusing on training that sparse subnetwork by re-using the same set. We find our coordinated framework to offer orthogonal gains to existing real image data augmentation methods, and we additionally present a new feature-level augmentation that can be applied together with them. Comprehensive experiments endorse the effectiveness of our proposed framework, across various GAN architectures (SNGAN, BigGAN, and StyleGAN-V2) and diverse datasets (CIFAR-10, CIFAR-100, Tiny-ImageNet, ImageNet, and multiple few-shot generation datasets). Codes are available at: https://github.com/VITA-Group/Ultra-Data-Efficient-GAN-Training.

arXiv commentNeurIPS 21

References in corpus (44)

• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Spectral Normalization for Generative Adversarial Networks
• Self-Attention Generative Adversarial Networks
• Rethinking the Value of Network Pruning
• Theoretically Principled Trade-off between Robustness and Accuracy
• Which Training Methods for GANs do actually Converge?
• Data Augmentation Generative Adversarial Networks
• Adversarially Robust Generalization Requires More Data
• Big Self-Supervised Models are Strong Semi-Supervised Learners
• The State of Sparsity in Deep Neural Networks
• Few-Shot Learning with Graph Neural Networks
• MMD GAN: Towards Deeper Understanding of Moment Matching Network
• Robustness May Be at Odds with Accuracy
• FreeLB: Enhanced Adversarial Training for Natural Language Understanding
• A Review on Generative Adversarial Networks: Algorithms, Theory, and Applications
• Comparing Rewinding and Fine-tuning in Neural Network Pruning
• Picking Winning Tickets Before Training by Preserving Gradient Flow
• Adversarial Training Can Hurt Generalization
• Freeze the Discriminator: a Simple Baseline for Fine-Tuning GANs
• Playing the lottery with rewards and multiple languages: lottery tickets in RL and NLP
• ContraGAN: Contrastive Learning for Conditional Image Generation
• High-Fidelity Image Generation With Fewer Labels
• Adversarial Robustness May Be at Odds With Simplicity
• On Leveraging Pretrained GANs for Generation with Limited Data
• Improving Neural Language Modeling via Adversarial Training
• Few-Shot Adaptation of Generative Adversarial Networks
• The Early Phase of Neural Network Training
• Winning the Lottery with Continuous Sparsification
• Triple Wins: Boosting Accuracy, Robustness and Efficiency Together by Enabling Input-Adaptive Inference
• The Lottery Tickets Hypothesis for Supervised and Self-supervised Pre-training in Computer Vision Models
• The Difficulty of Training Sparse Neural Networks
• A Unified Lottery Ticket Hypothesis for Graph Neural Networks
• Improved Sample Complexities for Deep Networks and Robust Classification via an All-Layer Margin
• Improved Consistency Regularization for GANs
• Few-shot Image Generation with Elastic Weight Consolidation
• Transferring GANs: generating images from limited data
• Once-for-All Adversarial Training: In-Situ Tradeoff between Robustness and Accuracy for Free
• EarlyBERT: Efficient BERT Training via Early-bird Lottery Tickets
• Towards Understanding Adversarial Examples Systematically: Exploring Data Size, Task and Model Factors
• Adversarial Feature Augmentation and Normalization for Visual Recognition
• Efficient Lottery Ticket Finding: Less Data is More
• GANs Can Play Lottery Tickets Too