Implicit Semantic Data Augmentation for Deep Networks

Abstract

In this paper, we propose a novel implicit semantic data augmentation (ISDA) approach to complement traditional augmentation techniques like flipping, translation or rotation. Our work is motivated by the intriguing property that deep networks are surprisingly good at linearizing features, such that certain directions in the deep feature space correspond to meaningful semantic transformations, e.g., adding sunglasses or changing backgrounds. As a consequence, translating training samples along many semantic directions in the feature space can effectively augment the dataset to improve generalization. To implement this idea effectively and efficiently, we first perform an online estimate of the covariance matrix of deep features for each class, which captures the intra-class semantic variations. Then random vectors are drawn from a zero-mean normal distribution with the estimated covariance to augment the training data in that class. Importantly, instead of augmenting the samples explicitly, we can directly minimize an upper bound of the expected cross-entropy (CE) loss on the augmented training set, leading to a highly efficient algorithm. In fact, we show that the proposed ISDA amounts to minimizing a novel robust CE loss, which adds negligible extra computational cost to a normal training procedure. Although being simple, ISDA consistently improves the generalization performance of popular deep models (ResNets and DenseNets) on a variety of datasets, e.g., CIFAR-10, CIFAR-100 and ImageNet. Code for reproducing our results is available at https://github.com/blackfeather-wang/ISDA-for-Deep-Networks.

Accepted by NeurIPS 2019

Cited by in corpus (17)

• Free Lunch for Few-shot Learning: Distribution Calibration
• COSTA: Covariance-Preserving Feature Augmentation for Graph Contrastive Learning
• Deep Long-Tailed Learning: A Survey
• Dynamic Neural Networks: A Survey
• Data Augmentation Revisited: Rethinking the Distribution Gap between Clean and Augmented Data
• On Feature Normalization and Data Augmentation
• Revisiting Locally Supervised Learning: an Alternative to End-to-end Training
• Regularizing Deep Networks with Semantic Data Augmentation
• FASA: Feature Augmentation and Sampling Adaptation for Long-Tailed Instance Segmentation
• Label-Aware Distribution Calibration for Long-tailed Classification
• On the Pitfalls of Learning with Limited Data: A Facial Expression Recognition Case Study
• Adaptive Focus for Efficient Video Recognition
• Repaint: Improving the Generalization of Down-Stream Visual Tasks by Generating Multiple Instances of Training Examples
• Data Augmentation via Structured Adversarial Perturbations
• Compensation Learning
• Transferable Semantic Augmentation for Domain Adaptation
• Robustness and Overfitting Behavior of Implicit Background Models