Puzzle Mix: Exploiting Saliency and Local Statistics for Optimal Mixup

Abstract

While deep neural networks achieve great performance on fitting the training distribution, the learned networks are prone to overfitting and are susceptible to adversarial attacks. In this regard, a number of mixup based augmentation methods have been recently proposed. However, these approaches mainly focus on creating previously unseen virtual examples and can sometimes provide misleading supervisory signal to the network. To this end, we propose Puzzle Mix, a mixup method for explicitly utilizing the saliency information and the underlying statistics of the natural examples. This leads to an interesting optimization problem alternating between the multi-label objective for optimal mixing mask and saliency discounted optimal transport objective. Our experiments show Puzzle Mix achieves the state of the art generalization and the adversarial robustness results compared to other mixup methods on CIFAR-100, Tiny-ImageNet, and ImageNet datasets. The source code is available at https://github.com/snu-mllab/PuzzleMix.

arXiv commentPublished at ICML 2020

Cited by in corpus (22)

• An overview of mixing augmentation methods and augmentation strategies
• Deep Weakly-Supervised Learning Methods for Classification and Localization in Histology Images: A Survey
• ResizeMix: Mixing Data with Preserved Object Information and True Labels
• How Does Mixup Help With Robustness and Generalization?
• GuidedMixup: An Efficient Mixup Strategy Guided by Saliency Maps
• Deep Learning to Estimate Permeability using Geophysical Data
• Co-Mixup: Saliency Guided Joint Mixup with Supermodular Diversity
• Object-aware Contrastive Learning for Debiased Scene Representation
• Boosting Discriminative Visual Representation Learning with Scenario-Agnostic Mixup
• It Takes Two to Tango: Mixup for Deep Metric Learning
• When and How Mixup Improves Calibration
• MixSiam: A Mixture-based Approach to Self-supervised Representation Learning
• Towards Understanding the Data Dependency of Mixup-style Training
• Robust Semantic Segmentation with Superpixel-Mix
• Distilling Ensemble of Explanations for Weakly-Supervised Pre-Training of Image Segmentation Models
• UPANets: Learning from the Universal Pixel Attention Networks
• StackMix: A complementary Mix algorithm
• StyleAugment: Learning Texture De-biased Representations by Style Augmentation without Pre-defined Textures
• k-Mixup Regularization for Deep Learning via Optimal Transport
• TransMix: Attend to Mix for Vision Transformers
• Why Mixup Improves the Model Performance
• Noisy Feature Mixup