Hypernetwork-Based Augmentation
arXiv:2006.06320
Abstract
Data augmentation is an effective technique to improve the generalization of deep neural networks. Recently, AutoAugment proposed a well-designed search space and a search algorithm that automatically finds augmentation policies in a data-driven manner. However, AutoAugment is computationally intensive. In this paper, we propose an efficient gradient-based search algorithm, called Hypernetwork-Based Augmentation (HBA), which simultaneously learns model parameters and augmentation hyperparameters in a single training. Our HBA uses a hypernetwork to approximate a population-based training algorithm, which enables us to tune augmentation hyperparameters by gradient descent. Besides, we introduce a weight sharing strategy that simplifies our hypernetwork architecture and speeds up our search algorithm. We conduct experiments on CIFAR-10, CIFAR-100, SVHN, and ImageNet. Our results show that HBA is competitive to the state-of-the-art methods in terms of both search speed and accuracy.
References in corpus (13)
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- Neural Architecture Search with Reinforcement Learning
- Improved Regularization of Convolutional Neural Networks with Cutout
- Unsupervised Data Augmentation for Consistency Training
- DARTS: Differentiable Architecture Search
- Efficient Neural Architecture Search via Parameter Sharing
- Hyper-Parameter Optimization: A Review of Algorithms and Applications
- Shake-Shake regularization
- ShakeDrop Regularization for Deep Residual Learning
- Population Based Augmentation: Efficient Learning of Augmentation Policy Schedules
- Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels
- Self-Tuning Networks: Bilevel Optimization of Hyperparameters using Structured Best-Response Functions