An Empirical Evaluation on Robustness and Uncertainty of Regularization Methods

Abstract

Despite apparent human-level performances of deep neural networks (DNN), they behave fundamentally differently from humans. They easily change predictions when small corruptions such as blur and noise are applied on the input (lack of robustness), and they often produce confident predictions on out-of-distribution samples (improper uncertainty measure). While a number of researches have aimed to address those issues, proposed solutions are typically expensive and complicated (e.g. Bayesian inference and adversarial training). Meanwhile, many simple and cheap regularization methods have been developed to enhance the generalization of classifiers. Such regularization methods have largely been overlooked as baselines for addressing the robustness and uncertainty issues, as they are not specifically designed for that. In this paper, we provide extensive empirical evaluations on the robustness and uncertainty estimates of image classifiers (CIFAR-100 and ImageNet) trained with state-of-the-art regularization methods. Furthermore, experimental results show that certain regularization methods can serve as strong baseline methods for robustness and uncertainty estimation of DNNs.

Accepted at ICML 2019 Workshop on Uncertainty and Robustness in Deep Learning. 7 pages, 1 figure

References in corpus (21)

• Adam: A Method for Stochastic Optimization
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• mixup: Beyond Empirical Risk Minimization
• What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?
• Improved Regularization of Convolutional Neural Networks with Cutout
• On Calibration of Modern Neural Networks
• A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks
• LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop
• Random Erasing Data Augmentation
• AutoAugment: Learning Augmentation Policies from Data
• CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features
• DropBlock: A regularization method for convolutional networks
• Gather-Excite: Exploiting Feature Context in Convolutional Neural Networks
• Generalisation in humans and deep neural networks
• Shake-Shake regularization
• ShakeDrop Regularization for Deep Residual Learning
• NSML: Meet the MLaaS platform with a real-world case study
• Attacking the Madry Defense Model with $L_1$-based Adversarial Examples
• Feature Denoising for Improving Adversarial Robustness
• NSML: A Machine Learning Platform That Enables You to Focus on Your Models

Cited by in corpus (10)

• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
• AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty
• Evaluating Prediction-Time Batch Normalization for Robustness under Covariate Shift
• AdamP: Slowing Down the Slowdown for Momentum Optimizers on Scale-invariant Weights
• Self-Knowledge Distillation with Progressive Refinement of Targets
• Analyzing Effects of Mixed Sample Data Augmentation on Model Interpretability
• StyleAugment: Learning Texture De-biased Representations by Style Augmentation without Pre-defined Textures
• Improved Robustness to Open Set Inputs via Tempered Mixup
• Learning from Matured Dumb Teacher for Fine Generalization
• Frustratingly Easy Uncertainty Estimation for Distribution Shift