Understanding and Improving Robustness of Vision Transformers through Patch-based Negative Augmentation
arXiv:2110.07858
Abstract
We investigate the robustness of vision transformers (ViTs) through the lens of their special patch-based architectural structure, i.e., they process an image as a sequence of image patches. We find that ViTs are surprisingly insensitive to patch-based transformations, even when the transformation largely destroys the original semantics and makes the image unrecognizable by humans. This indicates that ViTs heavily use features that survived such transformations but are generally not indicative of the semantic class to humans. Further investigations show that these features are useful but non-robust, as ViTs trained on them can achieve high in-distribution accuracy, but break down under distribution shifts. From this understanding, we ask: can training the model to rely less on these features improve ViT robustness and out-of-distribution performance? We use the images transformed with our patch-based operations as negatively augmented views and offer losses to regularize the training away from using non-robust features. This is a complementary view to existing research that mostly focuses on augmenting inputs with semantic-preserving transformations to enforce models' invariance. We show that patch-based negative augmentation consistently improves robustness of ViTs across a wide set of ImageNet based robustness benchmarks. Furthermore, we find our patch-based negative augmentation are complementary to traditional (positive) data augmentation, and together boost the performance further.
arXiv commentAccepted to NeurIPS-2022
References in corpus (10)
- Learning deep representations by mutual information estimation and maximization
- Theoretically Principled Trade-off between Robustness and Accuracy
- ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness
- Some Improvements on Deep Convolutional Neural Network Based Image Classification
- Intriguing Properties of Vision Transformers
- Natural Adversarial Examples
- Approximating CNNs with Bag-of-local-Features models works surprisingly well on ImageNet
- How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers
- The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization
- Negative Data Augmentation