Using Self-Supervised Learning Can Improve Model Robustness and Uncertainty

Abstract

Self-supervision provides effective representations for downstream tasks without requiring labels. However, existing approaches lag behind fully supervised training and are often not thought beneficial beyond obviating or reducing the need for annotations. We find that self-supervision can benefit robustness in a variety of ways, including robustness to adversarial examples, label corruption, and common input corruptions. Additionally, self-supervision greatly benefits out-of-distribution detection on difficult, near-distribution outliers, so much so that it exceeds the performance of fully supervised methods. These results demonstrate the promise of self-supervision for improving robustness and uncertainty estimation and establish these tasks as new axes of evaluation for future self-supervised learning research.

arXiv commentNeurIPS 2019; code and data available at https://github.com/hendrycks/ss-ood

References in corpus (14)

• Representation Learning with Contrastive Predictive Coding
• Learning deep representations by mutual information estimation and maximization
• SGDR: Stochastic Gradient Descent with Warm Restarts
• Focal Loss for Dense Object Detection
• Theoretically Principled Trade-off between Robustness and Accuracy
• Data-Efficient Image Recognition with Contrastive Predictive Coding
• Adversarially Robust Generalization Requires More Data
• Using Pre-Training Can Improve Model Robustness and Uncertainty
• Deep Anomaly Detection with Outlier Exposure
• Deep Anomaly Detection Using Geometric Transformations
• Adversarial Risk and the Dangers of Evaluating Against Weak Attacks
• Adversarial Machine Learning at Scale
• Benchmarking Neural Network Robustness to Common Corruptions and Perturbations
• Feature Denoising for Improving Adversarial Robustness

Cited by in corpus (33)

• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
• Self-Supervised Representation Learning: Introduction, Advances and Challenges
• Label-Efficient Self-Supervised Federated Learning for Tackling Data Heterogeneity in Medical Imaging
• Adaptive Extreme Edge Computing for Wearable Devices
• Video Transformers: A Survey
• DeepSMILE: Contrastive self-supervised pre-training benefits MSI and HRD classification directly from H&E whole-slide images in colorectal and breast cancer
• Outlier Exposure with Confidence Control for Out-of-Distribution Detection
• Self-Supervised Anomaly Detection in Computer Vision and Beyond: A Survey and Outlook
• Unsupervised Time Series Outlier Detection with Diversity-Driven Convolutional Ensembles -- Extended Version
• Quasi Anomalous Knowledge: Searching for new physics with embedded knowledge
• Enhancing Network Initialization for Medical AI Models Using Large-Scale, Unlabeled Natural Images
• An Iterative Method for Unsupervised Robust Anomaly Detection Under Data Contamination
• Improving colonoscopy lesion classification using semi-supervised deep learning
• Self-supervision for medical image classification: state-of-the-art performance with ~100 labeled training samples per class
• FAIR: Frequency-aware Image Restoration for Industrial Visual Anomaly Detection
• Reconstruction from edge image combined with color and gradient difference for industrial surface anomaly detection
• Self-supervised Representation Learning With Path Integral Clustering For Speaker Diarization
• A graph neural network-based model with Out-of-Distribution Robustness for enhancing Antiretroviral Therapy Outcome Prediction for HIV-1
• How Robust are Discriminatively Trained Zero-Shot Learning Models?
• Self-Supervised Learning for Medical Image Data with Anatomy-Oriented Imaging Planes
• MusiCoder: A Universal Music-Acoustic Encoder Based on Transformers
• Understanding Open-Set Recognition by Jacobian Norm and Inter-Class Separation
• Robust Image Classification Using A Low-Pass Activation Function and DCT Augmentation
• Towards Trustworthy Dataset Distillation
• Towards Human-Interpretable Prototypes for Visual Assessment of Image Classification Models
• On Calibration of Mixup Training for Deep Neural Networks
• Understanding the limitations of self-supervised learning for tabular anomaly detection
• Improving the Diversity of Bootstrapped DQN by Replacing Priors With Noise
• Good-looking but Lacking Faithfulness: Understanding Local Explanation Methods through Trend-based Testing
• Smooth Imitation Learning via Smooth Costs and Smooth Policies
• Clarifying Myths About the Relationship Between Shape Bias, Accuracy, and Robustness
• Label-free Monitoring of Self-Supervised Learning Progress
• Visual Probing: Cognitive Framework for Explaining Self-Supervised Image Representations