Using Pre-Training Can Improve Model Robustness and Uncertainty

Abstract

He et al. (2018) have called into question the utility of pre-training by showing that training from scratch can often yield similar performance to pre-training. We show that although pre-training may not improve performance on traditional classification metrics, it improves model robustness and uncertainty estimates. Through extensive experiments on adversarial examples, label corruption, class imbalance, out-of-distribution detection, and confidence calibration, we demonstrate large gains from pre-training and complementary effects with task-specific methods. We introduce adversarial pre-training and show approximately a 10% absolute improvement over the previous state-of-the-art in adversarial robustness. In some cases, using pre-training without task-specific methods also surpasses the state-of-the-art, highlighting the need for pre-training when evaluating future methods on robustness and uncertainty tasks.

arXiv commentICML 2019. PyTorch code here: https://github.com/hendrycks/pre-training Figure 3 updated

Cited by in corpus (141)

• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
• On the Opportunities and Risks of Foundation Models
• Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks
• AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty
• Image Classification with Deep Learning in the Presence of Noisy Labels: A Survey
• Using Self-Supervised Learning Can Improve Model Robustness and Uncertainty
• Large-Scale Adversarial Training for Vision-and-Language Representation Learning
• CSI: Novelty Detection via Contrastive Learning on Distributionally Shifted Instances
• Provably Robust Deep Learning via Adversarially Trained Smoothed Classifiers
• A Primer on Motion Capture with Deep Learning: Principles, Pitfalls and Perspectives
• Invisible for both Camera and LiDAR: Security of Multi-Sensor Fusion based Perception in Autonomous Driving Under Physical-World Attacks
• Adversarial Weight Perturbation Helps Robust Generalization
• Strategies for Pre-training Graph Neural Networks
• Are Labels Required for Improving Adversarial Robustness?
• Uncovering the Limits of Adversarial Training against Norm-Bounded Adversarial Examples
• Adversarial Attack Vulnerability of Medical Image Analysis Systems: Unexplored Factors
• Reinventing 2D Convolutions for 3D Images
• RobustBench: a standardized adversarial robustness benchmark
• Exploring the Limits of Out-of-Distribution Detection
• Fixing Data Augmentation to Improve Adversarial Robustness
• Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awareness
• A Survey of Label-noise Representation Learning: Past, Present and Future
• MMA Training: Direct Input Space Margin Maximization through Adversarial Training
• Unsolved Problems in ML Safety
• Verified Uncertainty Calibration
• Exchanging Dual Encoder-Decoder: A New Strategy for Change Detection with Semantic Guidance and Spatial Localization
• Adversarial Self-Supervised Contrastive Learning
• Self-Supervised Representation Learning for Astronomical Images
• Deep Long-Tailed Learning: A Survey
• Does Your Dermatology Classifier Know What It Doesn't Know? Detecting the Long-Tail of Unseen Conditions
• A Simple Fix to Mahalanobis Distance for Improving Near-OOD Detection
• Test-Time Training with Self-Supervision for Generalization under Distribution Shifts
• Unlabeled Data Improves Adversarial Robustness
• Boosting Adversarial Training with Hypersphere Embedding
• Towards Evaluating the Robustness of Deep Diagnostic Models by Adversarial Attack
• Parameters Estimation for the Cosmic Microwave Background with Bayesian Neural Networks
• Practical Solutions for Machine Learning Safety in Autonomous Vehicles
• Understanding Softmax Confidence and Uncertainty
• Understanding and Improving Fast Adversarial Training
• Recent Advances in Adversarial Training for Adversarial Robustness
• On the Loss Landscape of Adversarial Training: Identifying Challenges and How to Overcome Them
• Do Wider Neural Networks Really Help Adversarial Robustness?
• Adversarial Distributional Training for Robust Deep Learning
• Learning with Feature-Dependent Label Noise: A Progressive Approach
• HYDRA: Pruning Adversarially Robust Neural Networks
• Geometry-aware Instance-reweighted Adversarial Training
• Randomized Smoothing of All Shapes and Sizes
• Adversarial Robustness via Fisher-Rao Regularization
• Pretrained Transformers Improve Out-of-Distribution Robustness
• BERT & Family Eat Word Salad: Experiments with Text Understanding
• Two Sides of the Same Coin: White-box and Black-box Attacks for Transfer Learning
• Adversarial Neural Pruning with Latent Vulnerability Suppression
• On Fast Adversarial Robustness Adaptation in Model-Agnostic Meta-Learning
• How does Early Stopping Help Generalization against Label Noise?
• Evaluating the Robustness of Self-Supervised Learning in Medical Imaging
• On the effectiveness of adversarial training against common corruptions
• Adversarial training in communication constrained federated learning
• Intriguing properties of adversarial training at scale
• Dynamic GNNs for Precise Seizure Detection and Classification from EEG Data
• Exploring Memorization in Adversarial Training
• Hyperparameter-Free Out-of-Distribution Detection Using Softmax of Scaled Cosine Similarity
• Attacking Adversarial Attacks as A Defense
• FREE: Feature Refinement for Generalized Zero-Shot Learning
• Bag of Tricks for Adversarial Training
• Learnable Boundary Guided Adversarial Training
• How benign is benign overfitting?
• Contrastive Learning Improves Model Robustness Under Label Noise
• On the Generalization Properties of Adversarial Training
• Manifold Regularization for Locally Stable Deep Neural Networks
• Efficient Adversarial Training with Transferable Adversarial Examples
• An Incremental Learning framework for Large-scale CTR Prediction
• Understanding Generalization in Adversarial Training via the Bias-Variance Decomposition
• An Empirical Study on Robustness to Spurious Correlations using Pre-trained Language Models
• Adversarial Robustness under Long-Tailed Distribution
• In-N-Out: Pre-Training and Self-Training using Auxiliary Information for Out-of-Distribution Robustness
• Multi-Stage Conversational Passage Retrieval: An Approach to Fusing Term Importance Estimation and Neural Query Rewriting
• Robust Training with Ensemble Consensus
• Improved OOD Generalization via Adversarial Training and Pre-training
• Imbalanced Gradients: A Subtle Cause of Overestimated Adversarial Robustness
• Adversarial Robustness through the Lens of Convolutional Filters
• DiCOVA-Net: Diagnosing COVID-19 using Acoustics based on Deep Residual Network for the DiCOVA Challenge 2021
• When Human Pose Estimation Meets Robustness: Adversarial Algorithms and Benchmarks
• Combining Self-Supervised and Supervised Learning with Noisy Labels
• Guided Interpolation for Adversarial Training
• $k$Folden: $k$-Fold Ensemble for Out-Of-Distribution Detection
• On the Robustness of Human Pose Estimation
• How and When Adversarial Robustness Transfers in Knowledge Distillation?
• Revisiting Classical Bagging with Modern Transfer Learning for On-the-fly Disaster Damage Detector
• Selective Question Answering under Domain Shift
• TRIER: Template-Guided Neural Networks for Robust and Interpretable Sleep Stage Identification from EEG Recordings
• A Fine-Grained Analysis on Distribution Shift
• Towards Evaluating the Robustness of Neural Networks Learned by Transduction
• Data Quality Matters For Adversarial Training: An Empirical Study
• On Calibration of Mixup Training for Deep Neural Networks
• A comprehensive study on the prediction reliability of graph neural networks for virtual screening
• Adversarial Training Helps Transfer Learning via Better Representations
• Practical Evaluation of Out-of-Distribution Detection Methods for Image Classification
• Toward High Quality Facial Representation Learning
• Improving Adversarial Robustness via Unlabeled Out-of-Domain Data
• Improving Uncertainty Calibration of Deep Neural Networks via Truth Discovery and Geometric Optimization
• LAFEAT: Piercing Through Adversarial Defenses with Latent Features
• Rejuvenating Low-Frequency Words: Making the Most of Parallel Data in Non-Autoregressive Translation
• Learning with Noisy Labels by Efficient Transition Matrix Estimation to Combat Label Miscorrection
• Improving Model Robustness with Latent Distribution Locally and Globally
• Improving Neural Machine Translation by Bidirectional Training
• Learning Stable Classifiers by Transferring Unstable Features
• Bridging In- and Out-of-distribution Samples for Their Better Discriminability
• Multi-stage Optimization based Adversarial Training
• Adversarial Robustness for Unsupervised Domain Adaptation
• Natural Perturbed Training for General Robustness of Neural Network Classifiers
• Meta-Learning the Search Distribution of Black-Box Random Search Based Adversarial Attacks
• What it Thinks is Important is Important: Robustness Transfers through Input Gradients
• Cross-denoising Network against Corrupted Labels in Medical Image Segmentation with Domain Shift
• Towards to Robust and Generalized Medical Image Segmentation Framework
• Learning to Profile: User Meta-Profile Network for Few-Shot Learning
• IM-META: Influence Maximization Using Node Metadata in Networks With Unknown Topology
• Robustness and Reliability When Training With Noisy Labels
• Selection for short-term empowerment accelerates the evolution of homeostatic neural cellular automata
• Enhance Diffusion to Improve Robust Generalization
• On the Robustness of Pretraining and Self-Supervision for a Deep Learning-based Analysis of Diabetic Retinopathy
• Adversarial Attacks on ML Defense Models Competition
• Model-Agnostic Meta-Attack: Towards Reliable Evaluation of Adversarial Robustness
• Dynamic Defense Approach for Adversarial Robustness in Deep Neural Networks via Stochastic Ensemble Smoothed Model
• The USYD-JD Speech Translation System for IWSLT 2021
• Removing Undesirable Feature Contributions Using Out-of-Distribution Data
• Improving Reverberant Speech Separation with Multi-stage Training and Curriculum Learning
• Out-of-Scope Intent Detection with Self-Supervision and Discriminative Training
• Feature Distillation With Guided Adversarial Contrastive Learning
• ImageNet Pre-training also Transfers Non-Robustness
• A Computer Vision Approach to Combat Lyme Disease
• Frustratingly Easy Uncertainty Estimation for Distribution Shift
• Knowing More About Questions Can Help: Improving Calibration in Question Answering
• Exploring Novel Pooling Strategies for Edge Preserved Feature Maps in Convolutional Neural Networks
• Adversarial Training with Stochastic Weight Average
• Training Provably Robust Models by Polyhedral Envelope Regularization
• Identifying and Exploiting Structures for Reliable Deep Learning
• Out-of-Distribution Example Detection in Deep Neural Networks using Distance to Modelled Embedding
• Luring of transferable adversarial perturbations in the black-box paradigm
• On Deep Neural Network Calibration by Regularization and its Impact on Refinement
• Semi-Supervised Text Classification via Self-Pretraining
• Characterizing and Improving the Robustness of Self-Supervised Learning through Background Augmentations