Benchmarking Neural Network Robustness to Common Corruptions and Surface Variations

Abstract

In this paper we establish rigorous benchmarks for image classifier robustness. Our first benchmark, ImageNet-C, standardizes and expands the corruption robustness topic, while showing which classifiers are preferable in safety-critical applications. Unlike recent robustness research, this benchmark evaluates performance on commonplace corruptions not worst-case adversarial corruptions. We find that there are negligible changes in relative corruption robustness from AlexNet to ResNet classifiers, and we discover ways to enhance corruption robustness. Then we propose a new dataset called Icons-50 which opens research on a new kind of robustness, surface variation robustness. With this dataset we evaluate the frailty of classifiers on new styles of known objects and unexpected instances of known classes. We also demonstrate two methods that improve surface variation robustness. Together our benchmarks may aid future work toward networks that learn fundamental class structure and also robustly generalize.

Superseded by _Benchmarking Neural Network Robustness to Common Corruptions and Perturbations_ arXiv:1903.12261

References in corpus (17)

• Robust Physical-World Attacks on Deep Learning Models
• Adversarial Examples Are Not Easily Detected: Bypassing Ten Detection Methods
• Shake-Shake regularization
• Defensive Distillation is Not Robust to Adversarial Examples
• Multi-Scale Dense Networks for Resource Efficient Image Classification
• Why do deep convolutional networks generalize so poorly to small image transformations?
• Comparing deep neural networks against humans: object recognition when the signal gets weaker
• Do CIFAR-10 Classifiers Generalize to CIFAR-10?
• Early Methods for Detecting Adversarial Images
• Towards Proving the Adversarial Robustness of Deep Neural Networks
• Using Trusted Data to Train Deep Networks on Labels Corrupted by Severe Noise
• CondenseNet: An Efficient DenseNet using Learned Group Convolutions
• Certified Defenses for Data Poisoning Attacks
• Standard detectors aren't (currently) fooled by physical adversarial stop signs
• Open Category Detection with PAC Guarantees
• Quality Resilient Deep Neural Networks
• A Study and Comparison of Human and Deep Learning Recognition Performance Under Visual Distortions

Cited by in corpus (41)

• On Evaluating Adversarial Robustness
• Adversarial Examples Are Not Bugs, They Are Features
• Adversarial Examples Are a Natural Consequence of Test Error in Noise
• The Origins and Prevalence of Texture Bias in Convolutional Neural Networks
• Outlier Exposure with Confidence Control for Out-of-Distribution Detection
• Improving Robustness Without Sacrificing Accuracy with Patch Gaussian Augmentation
• Testing Robustness Against Unforeseen Adversaries
• Self-Supervised Policy Adaptation during Deployment
• Affinity and Diversity: Quantifying Mechanisms of Data Augmentation
• Adversarial Examples Improve Image Recognition
• Attribute Restoration Framework for Anomaly Detection
• Greedy Policy Search: A Simple Baseline for Learnable Test-Time Augmentation
• Using Videos to Evaluate Image Model Robustness
• Unadversarial Examples: Designing Objects for Robust Vision
• BREEDS: Benchmarks for Subpopulation Shift
• SECANT: Self-Expert Cloning for Zero-Shot Generalization of Visual Policies
• 3DB: A Framework for Debugging Computer Vision Models
• Robustifying Models Against Adversarial Attacks by Langevin Dynamics
• Using learned optimizers to make models robust to input noise
• Convergence and Margin of Adversarial Training on Separable Data
• LiftPool: Bidirectional ConvNet Pooling
• Negative Data Augmentation
• Respecting Domain Relations: Hypothesis Invariance for Domain Generalization
• Identifying Statistical Bias in Dataset Replication
• Adversarial Attacks against Deep Saliency Models
• Shift Equivariance in Object Detection
• Calibrated neighborhood aware confidence measure for deep metric learning
• On the relationship between class selectivity, dimensionality, and robustness
• Distance Matters For Improving Performance Estimation Under Covariate Shift
• Approximate Selection with Guarantees using Proxies
• Diminishing the Effect of Adversarial Perturbations via Refining Feature Representation
• Defective Convolutional Networks
• SI-Score: An image dataset for fine-grained analysis of robustness to object location, rotation and size
• Can Perceptual Guidance Lead to Semantically Explainable Adversarial Perturbations?
• Domain Invariant Adversarial Learning
• Applications of the Streaming Networks
• Reappraising Domain Generalization in Neural Networks
• Sparsifying and Down-scaling Networks to Increase Robustness to Distortions
• MUTE: Data-Similarity Driven Multi-hot Target Encoding for Neural Network Design
• Oriole: Thwarting Privacy against Trustworthy Deep Learning Models
• Linking average- and worst-case perturbation robustness via class selectivity and dimensionality