Negative Data Augmentation

Abstract

Data augmentation is often used to enlarge datasets with synthetic samples generated in accordance with the underlying data distribution. To enable a wider range of augmentations, we explore negative data augmentation strategies (NDA)that intentionally create out-of-distribution samples. We show that such negative out-of-distribution samples provide information on the support of the data distribution, and can be leveraged for generative modeling and representation learning. We introduce a new GAN training objective where we use NDA as an additional source of synthetic data for the discriminator. We prove that under suitable conditions, optimizing the resulting objective still recovers the true data distribution but can directly bias the generator towards avoiding samples that lack the desired structure. Empirically, models trained with our method achieve improved conditional/unconditional image generation along with improved anomaly detection capabilities. Further, we incorporate the same negative data augmentation strategy in a contrastive learning framework for self-supervised representation learning on images and videos, achieving improved performance on downstream image classification, object detection, and action recognition tasks. These results suggest that prior knowledge on what does not constitute valid data is an effective form of weak supervision across a range of unsupervised learning tasks.

Accepted at ICLR 2021

References in corpus (12)

• Distributed Representations of Words and Phrases and their Compositionality
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Improved Regularization of Convolutional Neural Networks with Cutout
• A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks
• Momentum Contrast for Unsupervised Visual Representation Learning
• Data Augmentation Generative Adversarial Networks
• Estimating divergence functionals and the likelihood ratio by convex risk minimization
• Deep Anomaly Detection with Outlier Exposure
• Benchmarking Neural Network Robustness to Common Corruptions and Surface Variations
• A critical analysis of self-supervision, or what we can learn from a single image
• Implicit Generation and Generalization in Energy-Based Models
• Unsupervised Adversarial Invariance

Cited by in corpus (3)

• Understanding and Improving Robustness of Vision Transformers through Patch-based Negative Augmentation
• Robust Learning Meets Generative Models: Can Proxy Distributions Improve Adversarial Robustness?
• OMASGAN: Out-of-Distribution Minimum Anomaly Score GAN for Sample Generation on the Boundary