Benchmarking Neural Network Robustness to Common Corruptions and Perturbations

Abstract

In this paper we establish rigorous benchmarks for image classifier robustness. Our first benchmark, ImageNet-C, standardizes and expands the corruption robustness topic, while showing which classifiers are preferable in safety-critical applications. Then we propose a new dataset called ImageNet-P which enables researchers to benchmark a classifier's robustness to common perturbations. Unlike recent robustness research, this benchmark evaluates performance on common corruptions and perturbations not worst-case adversarial perturbations. We find that there are negligible changes in relative corruption robustness from AlexNet classifiers to ResNet classifiers. Afterward we discover ways to enhance corruption and perturbation robustness. We even find that a bypassed adversarial defense provides substantial common perturbation robustness. Together our benchmarks may aid future work toward networks that robustly generalize.

arXiv commentICLR 2019 camera-ready; datasets available at https://github.com/hendrycks/robustness ; this article supersedes arXiv:1807.01697

Cited by in corpus (49)

• Shortcut Learning in Deep Neural Networks
• Using Self-Supervised Learning Can Improve Model Robustness and Uncertainty
• Multimodal Co-learning: Challenges, Applications with Datasets, Recent Advances and Future Directions
• Natural Adversarial Examples
• Why do deep convolutional networks generalize so poorly to small image transformations?
• Controversial stimuli: pitting neural networks against each other as models of human recognition
• RobustBench: a standardized adversarial robustness benchmark
• Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awareness
• Outlier Exposure with Confidence Control for Out-of-Distribution Detection
• MEMO: Test Time Robustness via Adaptation and Augmentation
• Testing Robustness Against Unforeseen Adversaries
• Towards Evaluating the Robustness of Deep Diagnostic Models by Adversarial Attack
• Unified Probabilistic Deep Continual Learning through Generative Replay and Open Set Recognition
• Hyperparameter Ensembles for Robustness and Uncertainty Quantification
• Exposing Previously Undetectable Faults in Deep Neural Networks
• Robustness Disparities in Commercial Face Detection
• Blind Image Deblurring with Unknown Kernel Size and Substantial Noise
• Perfect density models cannot guarantee anomaly detection
• Consistency Regularization for Certified Robustness of Smoothed Classifiers
• If a Human Can See It, So Should Your System: Reliability Requirements for Machine Vision Components
• How Good is Google Bard's Visual Understanding? An Empirical Study on Open Challenges
• On the effectiveness of adversarial training against common corruptions
• CheXphoto: 10,000+ Photos and Transformations of Chest X-rays for Benchmarking Deep Learning Robustness
• Infinitely Deep Bayesian Neural Networks with Stochastic Differential Equations
• CheXternal: Generalization of Deep Learning Models for Chest X-ray Interpretation to Photos of Chest X-rays and External Clinical Settings
• ACE: Adapting to Changing Environments for Semantic Segmentation
• An Unsupervised Information-Theoretic Perceptual Quality Metric
• Extending Environments To Measure Self-Reflection In Reinforcement Learning
• CEB Improves Model Robustness
• Deep Ensembles for Low-Data Transfer Learning
• Stationary Activations for Uncertainty Calibration in Deep Learning
• Bit Error Robustness for Energy-Efficient DNN Accelerators
• Denoised Internal Models: a Brain-Inspired Autoencoder against Adversarial Attacks
• Shifts: A Dataset of Real Distributional Shift Across Multiple Large-Scale Tasks
• CheXphotogenic: Generalization of Deep Learning Models for Chest X-ray Interpretation to Photos of Chest X-rays
• A Self-Supervised Feature Map Augmentation (FMA) Loss and Combined Augmentations Finetuning to Efficiently Improve the Robustness of CNNs
• Spatial Ensemble: a Novel Model Smoothing Mechanism for Student-Teacher Framework
• On Calibration of Mixup Training for Deep Neural Networks
• Compressive Visual Representations
• Utilizing Network Properties to Detect Erroneous Inputs
• Minimal Sufficient Views: A DNN model making predictions with more evidence has higher accuracy
• Golden Grain: Building a Secure and Decentralized Model Marketplace for MLaaS
• Uncover and Unlearn Nuisances: Agnostic Fully Test-Time Adaptation
• Partial Wasserstein and Maximum Mean Discrepancy distances for bridging the gap between outlier detection and drift detection
• RBUE: A ReLU-Based Uncertainty Estimation Method of Deep Neural Networks
• MDP Playground: An Analysis and Debug Testbed for Reinforcement Learning
• Exposing Semantic Segmentation Failures via Maximum Discrepancy Competition
• LocalNorm: Robust Image Classification through Dynamically Regularized Normalization
• Scene Uncertainty and the Wellington Posterior of Deterministic Image Classifiers