Robustness properties of Facebook's ResNeXt WSL models

Abstract

We investigate the robustness properties of ResNeXt class image recognition models trained with billion scale weakly supervised data (ResNeXt WSL models). These models, recently made public by Facebook AI, were trained with ~1B images from Instagram and fine-tuned on ImageNet. We show that these models display an unprecedented degree of robustness against common image corruptions and perturbations, as measured by the ImageNet-C and ImageNet-P benchmarks. They also achieve substantially improved accuracies on the recently introduced "natural adversarial examples" benchmark (ImageNet-A). The largest of the released models, in particular, achieves state-of-the-art results on ImageNet-C, ImageNet-P, and ImageNet-A by a large margin. The gains on ImageNet-C, ImageNet-P, and ImageNet-A far outpace the gains on ImageNet validation accuracy, suggesting the former as more useful benchmarks to measure further progress in image recognition. Remarkably, the ResNeXt WSL models even achieve a limited degree of adversarial robustness against state-of-the-art white-box attacks (10-step PGD attacks). However, in contrast to adversarially trained models, the robustness of the ResNeXt WSL models rapidly declines with the number of PGD steps, suggesting that these models do not achieve genuine adversarial robustness. Visualization of the learned features also confirms this conclusion. Finally, we show that although the ResNeXt WSL models are more shape-biased than comparable ImageNet-trained models in a shape-texture cue conflict experiment, they still remain much more texture-biased than humans, suggesting that they share some of the underlying characteristics of ImageNet-trained models that make this benchmark challenging.

arXiv comment10 pages, 4 figures, 4 tables; v5 corrects the ImageNet-A results and revises the discussion accordingly

References in corpus (7)

• Do ImageNet Classifiers Generalize to ImageNet?
• Natural Adversarial Examples
• Approximating CNNs with Bag-of-local-Features models works surprisingly well on ImageNet
• Improving Robustness Without Sacrificing Accuracy with Patch Gaussian Augmentation
• Feature Denoising for Improving Adversarial Robustness
• Evaluating and Understanding the Robustness of Adversarial Logit Pairing
• Adversarial Robustness as a Prior for Learned Representations

Cited by in corpus (11)

• Self-training with Noisy Student improves ImageNet classification
• Measuring Robustness to Natural Distribution Shifts in Image Classification
• Improving robustness against common corruptions by covariate shift adaptation
• MEMO: Test Time Robustness via Adaptation and Augmentation
• Testing Robustness Against Unforeseen Adversaries
• Self-supervised learning through the eyes of a child
• Pretrained Transformers Improve Out-of-Distribution Robustness
• On Interaction Between Augmentations and Corruptions in Natural Corruption Robustness
• Out-of-Scope Intent Detection with Self-Supervision and Discriminative Training
• Reading Isn't Believing: Adversarial Attacks On Multi-Modal Neurons
• Compositional generalization in semantic parsing with pretrained transformers