Regularizing Deep Networks with Semantic Data Augmentation

Abstract

Data augmentation is widely known as a simple yet surprisingly effective technique for regularizing deep networks. Conventional data augmentation schemes, e.g., flipping, translation or rotation, are low-level, data-independent and class-agnostic operations, leading to limited diversity for augmented samples. To this end, we propose a novel semantic data augmentation algorithm to complement traditional approaches. The proposed method is inspired by the intriguing property that deep networks are effective in learning linearized features, i.e., certain directions in the deep feature space correspond to meaningful semantic transformations, e.g., changing the background or view angle of an object. Based on this observation, translating training samples along many such directions in the feature space can effectively augment the dataset for more diversity. To implement this idea, we first introduce a sampling based method to obtain semantically meaningful directions efficiently. Then, an upper bound of the expected cross-entropy (CE) loss on the augmented training set is derived by assuming the number of augmented samples goes to infinity, yielding a highly efficient algorithm. In fact, we show that the proposed implicit semantic data augmentation (ISDA) algorithm amounts to minimizing a novel robust CE loss, which adds minimal extra computational cost to a normal training procedure. In addition to supervised learning, ISDA can be applied to semi-supervised learning tasks under the consistency regularization framework, where ISDA amounts to minimizing the upper bound of the expected KL-divergence between the augmented features and the original features. Although being simple, ISDA consistently improves the generalization performance of popular deep models (e.g., ResNets and DenseNets) on a variety of datasets, i.e., CIFAR-10, CIFAR-100, SVHN, ImageNet, and Cityscapes.

arXiv commentAccepted by IEEE Transactions on Pattern Analysis and Machine Intelligence (T-PAMI). Journal version of arXiv:1909.12220 (NeurIPS 2019). Code is available at https://github.com/blackfeather-wang/ISDA-for-Deep-Networks

References in corpus (30)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Auto-Encoding Variational Bayes
• Conditional Generative Adversarial Nets
• Rethinking Atrous Convolution for Semantic Image Segmentation
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?
• Improved Regularization of Convolutional Neural Networks with Cutout
• Conditional Image Synthesis With Auxiliary Classifier GANs
• Deep Learning Face Representation by Joint Identification-Verification
• SGDR: Stochastic Gradient Descent with Warm Restarts
• Unsupervised Data Augmentation for Consistency Training
• Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results
• Temporal Ensembling for Semi-Supervised Learning
• Training Very Deep Networks
• Interpolation Consistency Training for Semi-Supervised Learning
• MMDetection: Open MMLab Detection Toolbox and Benchmark
• Data Augmentation Generative Adversarial Networks
• Random Erasing Data Augmentation
• AutoAugment: Learning Augmentation Policies from Data
• Bayesian Convolutional Neural Networks with Bernoulli Approximate Variational Inference
• Multi-digit Number Recognition from Street View Imagery using Deep Convolutional Neural Networks
• Semi-Supervised Learning with Ladder Networks
• Large-Margin Softmax Loss for Convolutional Neural Networks
• GAN Augmentation: Augmenting Training Data using Generative Adversarial Networks
• Shake-Shake regularization
• Better Mixing via Deep Representations
• Implicit Semantic Data Augmentation for Deep Networks
• Convolutional Network for Attribute-driven and Identity-preserving Human Face Generation
• AttGAN: Facial Attribute Editing by Only Changing What You Want
• Revisiting Locally Supervised Learning: an Alternative to End-to-end Training

Cited by in corpus (2)

• Deep learning approach for identification of HII regions during reionization in 21-cm observations
• Sill-Net: Feature Augmentation with Separated Illumination Representation