Direct Differentiable Augmentation Search
arXiv:2104.04282
Abstract
Data augmentation has been an indispensable tool to improve the performance of deep neural networks, however the augmentation can hardly transfer among different tasks and datasets. Consequently, a recent trend is to adopt AutoML technique to learn proper augmentation policy without extensive hand-crafted tuning. In this paper, we propose an efficient differentiable search algorithm called Direct Differentiable Augmentation Search (DDAS). It exploits meta-learning with one-step gradient update and continuous relaxation to the expected training loss for efficient search. Our DDAS can achieve efficient augmentation search without relying on approximations such as Gumbel Softmax or second order gradient approximation. To further reduce the adverse effect of improper augmentations, we organize the search space into a two level hierarchy, in which we first decide whether to apply augmentation, and then determine the specific augmentation policy. On standard image classification benchmarks, our DDAS achieves state-of-the-art performance and efficiency tradeoff while reducing the search cost dramatically, e.g. 0.15 GPU hours for CIFAR-10. In addition, we also use DDAS to search augmentation for object detection task and achieve comparable performance with AutoAugment, while being 1000x faster.
arXiv commentICCV2021
References in corpus (18)
- Neural Architecture Search with Reinforcement Learning
- Categorical Reparameterization with Gumbel-Softmax
- Improved Regularization of Convolutional Neural Networks with Cutout
- Unsupervised Data Augmentation for Consistency Training
- DARTS: Differentiable Architecture Search
- Learning to Reweight Examples for Robust Deep Learning
- MMDetection: Open MMLab Detection Toolbox and Benchmark
- CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features
- Data Augmentation by Pairing Samples for Images Classification
- Shake-Shake regularization
- ReMixMatch: Semi-Supervised Learning with Distribution Alignment and Augmentation Anchoring
- Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels
- Stabilizing DARTS with Amended Gradient Estimation on Architectural Parameters
- Data Augmentation Revisited: Rethinking the Distribution Gap between Clean and Augmented Data
- Improving Auto-Augment via Augmentation-Wise Weight Sharing
- Meta Approach to Data Augmentation Optimization
- Data augmentation with Mobius transformations
- Hypernetwork-Based Augmentation