A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks

Abstract

Detecting test samples drawn sufficiently far away from the training distribution statistically or adversarially is a fundamental requirement for deploying a good classifier in many real-world machine learning applications. However, deep neural networks with the softmax classifier are known to produce highly overconfident posterior distributions even for such abnormal samples. In this paper, we propose a simple yet effective method for detecting any abnormal samples, which is applicable to any pre-trained softmax neural classifier. We obtain the class conditional Gaussian distributions with respect to (low- and upper-level) features of the deep models under Gaussian discriminant analysis, which result in a confidence score based on the Mahalanobis distance. While most prior methods have been evaluated for detecting either out-of-distribution or adversarial samples, but not both, the proposed method achieves the state-of-the-art performances for both cases in our experiments. Moreover, we found that our proposed method is more robust in harsh cases, e.g., when the training dataset has noisy labels or small number of samples. Finally, we show that the proposed method enjoys broader usage by applying it to class-incremental learning: whenever out-of-distribution samples are detected, our classification rule can incorporate new classes well without further training deep models.

Accepted in NIPS 2018

Cited by in corpus (273)

• FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence
• Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methods
• Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift
• CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features
• Energy-based Out-of-distribution Detection
• Generalized Out-of-Distribution Detection: A Survey
• Uncertainty Estimation Using a Single Deep Deterministic Neural Network
• CSI: Novelty Detection via Contrastive Learning on Distributionally Shifted Instances
• Learning Robust Global Representations by Penalizing Local Predictive Power
• Adversarial Example Detection for DNN Models: A Review and Experimental Comparison
• Scaling Out-of-Distribution Detection for Real-World Settings
• The Fishyscapes Benchmark: Measuring Blind Spots in Semantic Segmentation
• Input complexity and out-of-distribution detection with likelihood-based generative models
• GNNGuard: Defending Graph Neural Networks against Adversarial Attacks
• Zero-Shot Out-of-Distribution Detection Based on the Pre-trained Model CLIP
• Attack of the Tails: Yes, You Really Can Backdoor Federated Learning
• Exploring the Limits of Out-of-Distribution Detection
• A Review of Adversarial Attack and Defense for Classification Methods
• What Are Bayesian Neural Network Posteriors Really Like?
• Outlier Exposure with Confidence Control for Out-of-Distribution Detection
• The Conditional Entropy Bottleneck
• Self-Supervised Anomaly Detection in Computer Vision and Beyond: A Survey and Outlook
• Testing for Outliers with Conformal p-values
• How to Certify Machine Learning Based Safety-critical Systems? A Systematic Literature Review
• Does Your Dermatology Classifier Know What It Doesn't Know? Detecting the Long-Tail of Unseen Conditions
• Likelihood Regret: An Out-of-Distribution Detection Score For Variational Auto-encoder
• SegmentMeIfYouCan: A Benchmark for Anomaly Segmentation
• A Simple Fix to Mahalanobis Distance for Improving Near-OOD Detection
• Contrastive Out-of-Distribution Detection for Pretrained Transformers
• Detection of Face Recognition Adversarial Attacks
• Contrastive Training for Improved Out-of-Distribution Detection
• Inspect, Understand, Overcome: A Survey of Practical Methods for AI Safety
• Understanding Softmax Confidence and Uncertainty
• Reducing DNN Labelling Cost using Surprise Adequacy: An Industrial Case Study for Autonomous Driving
• Out-of-distribution Detection in Classifiers via Generation
• SSD: A Unified Framework for Self-Supervised Outlier Detection
• Why is the Mahalanobis Distance Effective for Anomaly Detection?
• Gradient Starvation: A Learning Proclivity in Neural Networks
• A Benchmark of Medical Out of Distribution Detection
• Unified Probabilistic Deep Continual Learning through Generative Replay and Open Set Recognition
• On the Importance of Gradients for Detecting Distributional Shifts in the Wild
• FINE Samples for Learning with Noisy Labels
• Probabilistic Modeling of Deep Features for Out-of-Distribution and Adversarial Detection
• A Causal View on Robustness of Neural Networks
• Confidence-Calibrated Adversarial Training: Generalizing to Unseen Attacks
• Can multi-label classification networks know what they don't know?
• Generalized ODIN: Detecting Out-of-distribution Image without Learning from Out-of-distribution Data
• Detecting Out-of-Distribution Inputs in Deep Neural Networks Using an Early-Layer Output
• Deep Verifier Networks: Verification of Deep Discriminative Models with Deep Generative Models
• Understanding Anomaly Detection with Deep Invertible Networks through Hierarchies of Distributions and Features
• How Does Fine-Tuning Impact Out-of-Distribution Detection for Vision-Language Models?
• Robust Out-of-distribution Detection for Neural Networks
• Learning on Graphs with Out-of-Distribution Nodes
• Learning Discriminative Representations and Decision Boundaries for Open Intent Detection
• Cautious Deep Learning
• Entropic Out-of-Distribution Detection: Seamless Detection of Unknown Examples
• Uncertainty as a Form of Transparency: Measuring, Communicating, and Using Uncertainty
• Feature Space Singularity for Out-of-Distribution Detection
• Perfect density models cannot guarantee anomaly detection
• Dataset Inference: Ownership Resolution in Machine Learning
• Self-Supervised Training Enhances Online Continual Learning
• Detecting Out-of-Distribution Examples with In-distribution Examples and Gram Matrices
• Lifelong Learning on Evolving Graphs Under the Constraints of Imbalanced Classes and New Classes
• A Less Biased Evaluation of Out-of-distribution Sample Detectors
• AdvFlow: Inconspicuous Black-box Adversarial Attacks using Normalizing Flows
• Learning Densities in Feature Space for Reliable Segmentation of Indoor Scenes
• A Unified Survey on Anomaly, Novelty, Open-Set, and Out-of-Distribution Detection: Solutions and Future Challenges
• Prior and Posterior Networks: A Survey on Evidential Deep Learning Methods For Uncertainty Estimation
• Multiobjective Evolutionary Pruning of Deep Neural Networks with Transfer Learning for improving their Performance and Robustness
• On Last-Layer Algorithms for Classification: Decoupling Representation from Uncertainty Estimation
• Distilling Causal Effect of Data in Class-Incremental Learning
• Statistical process monitoring of artificial neural networks
• Gaussian Mixture Proposals with Pull-Push Learning Scheme to Capture Diverse Events for Weakly Supervised Temporal Video Grounding
• Multimodal Detection of Unknown Objects on Roads for Autonomous Driving
• ML-LOO: Detecting Adversarial Examples with Feature Attribution
• Deflecting Adversarial Attacks
• Trust Issues: Uncertainty Estimation Does Not Enable Reliable OOD Detection On Medical Tabular Data
• Transfer Learning Gaussian Anomaly Detection by Fine-tuning Representations
• Adversarial Defense via Data Dependent Activation Function and Total Variation Minimization
• Understanding Open-Set Recognition by Jacobian Norm and Inter-Class Separation
• A First Look at Class Incremental Learning in Deep Learning Mobile Traffic Classification
• Two Souls in an Adversarial Image: Towards Universal Adversarial Example Detection using Multi-view Inconsistency
• Building robust classifiers through generation of confident out of distribution examples
• An Analysis of Adversarial Attacks and Defenses on Autonomous Driving Models
• Novelty Detection Via Blurring
• A Variational Dirichlet Framework for Out-of-Distribution Detection
• Detecting Backdoors in Neural Networks Using Novel Feature-Based Anomaly Detection
• Hyperparameter-Free Out-of-Distribution Detection Using Softmax of Scaled Cosine Similarity
• SPECTRE: Defending Against Backdoor Attacks Using Robust Statistics
• Lifelong Machine Learning with Deep Streaming Linear Discriminant Analysis
• DS-UI: Dual-Supervised Mixture of Gaussian Mixture Models for Uncertainty Inference
• Interpretability is a Kind of Safety: An Interpreter-based Ensemble for Adversary Defense
• The Hidden Uncertainty in a Neural Networks Activations
• Calibration of Model Uncertainty for Dropout Variational Inference
• Enhancing Consistency and Mitigating Bias: A Data Replay Approach for Incremental Learning
• Unified Classification and Rejection: A One-versus-All Framework
• Detecting Adversarial Examples Is (Nearly) As Hard As Classifying Them
• Autoregressive Models: What Are They Good For?
• Self-Supervised Features Improve Open-World Learning
• Unsupervised Out-of-Distribution Detection by Maximum Classifier Discrepancy
• ATOM: Robustifying Out-of-distribution Detection Using Outlier Mining
• Unsupervised Anomaly Detection with Adversarial Mirrored AutoEncoders
• FRODO: Free rejection of out-of-distribution samples: application to chest x-ray analysis
• Synthesize then Compare: Detecting Failures and Anomalies for Semantic Segmentation
• A Unified Benchmark for the Unknown Detection Capability of Deep Neural Networks
• DOCTOR: A Simple Method for Detecting Misclassification Errors
• Further Analysis of Outlier Detection with Deep Generative Models
• Are all outliers alike? On Understanding the Diversity of Outliers for Detecting OODs
• One Versus all for deep Neural Network Incertitude (OVNNI) quantification
• MOS: Towards Scaling Out-of-distribution Detection for Large Semantic Space
• Analysis of Confident-Classifiers for Out-of-distribution Detection
• Revisiting One-vs-All Classifiers for Predictive Uncertainty and Out-of-Distribution Detection in Neural Networks
• Anomalous Example Detection in Deep Learning: A Survey
• DeepLens: Interactive Out-of-distribution Data Detection in NLP Models
• Continual Learning using a Bayesian Nonparametric Dictionary of Weight Factors
• Likelihood Ratios for Out-of-Distribution Detection
• No True State-of-the-Art? OOD Detection Methods are Inconsistent across Datasets
• Effective and Robust Detection of Adversarial Examples via Benford-Fourier Coefficients
• Out-of-domain Detection for Natural Language Understanding in Dialog Systems
• Innovations Autoencoder and its Application in One-class Anomalous Sequence Detection
• A Critical Evaluation of Open-World Machine Learning
• Hierarchical VAEs Know What They Don't Know
• Detecting OODs as datapoints with High Uncertainty
• OpenCoS: Contrastive Semi-supervised Learning for Handling Open-set Unlabeled Data
• LiBRe: A Practical Bayesian Approach to Adversarial Detection
• Memory-Efficient Semi-Supervised Continual Learning: The World is its Own Replay Buffer
• Adaptation Across Extreme Variations using Unlabeled Domain Bridges
• Frequency Centric Defense Mechanisms against Adversarial Examples
• Detecting Adversarial Examples via Neural Fingerprinting
• Better the Devil you Know: An Analysis of Evasion Attacks using Out-of-Distribution Adversarial Examples
• Probing the Purview of Neural Networks via Gradient Analysis
• Extraction of Complex DNN Models: Real Threat or Boogeyman?
• Adversarial Artifact Detection in EEG-Based Brain-Computer Interfaces
• E-Stitchup: Data Augmentation for Pre-Trained Embeddings
• Efficacy of Pixel-Level OOD Detection for Semantic Segmentation
• A Data-driven Adversarial Examples Recognition Framework via Adversarial Feature Genome
• Detecting Adversarial Examples from Sensitivity Inconsistency of Spatial-Transform Domain
• ReSmooth: Detecting and Utilizing OOD Samples when Training with Data Augmentation
• OoDAnalyzer: Interactive Analysis of Out-of-Distribution Samples
• Do We Really Need to Learn Representations from In-domain Data for Outlier Detection?
• SQUID: Deep Feature In-Painting for Unsupervised Anomaly Detection
• GraN: An Efficient Gradient-Norm Based Detector for Adversarial and Misclassified Examples
• Pixel-wise Energy-biased Abstention Learning for Anomaly Segmentation on Complex Urban Driving Scenes
• A Unified Plug-and-Play Framework for Effective Data Denoising and Robust Abstention
• Mean-Field Approximation to Gaussian-Softmax Integral with Application to Uncertainty Estimation
• Random Directional Attack for Fooling Deep Neural Networks
• Regularizing Attention Networks for Anomaly Detection in Visual Question Answering
• Hybrid Generative-Contrastive Representation Learning
• SLA${}^2$P: Self-supervised Anomaly Detection with Adversarial Perturbation
• Uncertainty Quantification for Deep Context-Aware Mobile Activity Recognition and Unknown Context Discovery
• AED-PADA:Improving Generalizability of Adversarial Example Detection via Principal Adversarial Domain Adaptation
• Investigating Imperceptibility of Adversarial Attacks on Tabular Data: An Empirical Analysis
• Evaluating Robustness of Predictive Uncertainty Estimation: Are Dirichlet-based Models Reliable?
• Deep Learning and Traffic Classification: Lessons learned from a commercial-grade dataset with hundreds of encrypted and zero-day applications
• InFlow: Robust outlier detection utilizing Normalizing Flows
• On the Sample Complexity of Adversarial Multi-Source PAC Learning
• Practical Evaluation of Out-of-Distribution Detection Methods for Image Classification
• Calibrating Deep Neural Network Classifiers on Out-of-Distribution Datasets
• Multi-Loss Sub-Ensembles for Accurate Classification with Uncertainty Estimation
• Graceful Degradation and Related Fields
• Detecting Unknown DGAs without Context Information
• Advancing the Biological Plausibility and Efficacy of Hebbian Convolutional Neural Networks
• APP: Adaptive Prototypical Pseudo-Labeling for Few-shot OOD Detection
• Deep Variational Semi-Supervised Novelty Detection
• Out-of-Distribution Detection for Skin Lesion Images with Deep Isolation Forest
• A Simple Framework for Uncertainty in Contrastive Learning
• Provably Robust Detection of Out-of-distribution Data (almost) for free
• Bigeminal Priors Variational auto-encoder
• NGC: A Unified Framework for Learning with Open-World Noisy Data
• Drawing Robust Scratch Tickets: Subnetworks with Inborn Robustness Are Found within Randomly Initialized Networks
• Medical Aegis: Robust adversarial protectors for medical images
• Exploring Covariate and Concept Shift for Detection and Calibration of Out-of-Distribution Data
• Increasing Trustworthiness of Deep Neural Networks via Accuracy Monitoring
• Novelty Detection via Robust Variational Autoencoding
• Distance Matters For Improving Performance Estimation Under Covariate Shift
• Toward Metrics for Differentiating Out-of-Distribution Sets
• An Algorithm for Out-Of-Distribution Attack to Neural Network Encoder
• Loss Function Entropy Regularization for Diverse Decision Boundaries
• Fine-grained Uncertainty Modeling in Neural Networks
• Thinkback: Task-SpecificOut-of-Distribution Detection
• Enhanced Isotropy Maximization Loss: Seamless and High-Performance Out-of-Distribution Detection Simply Replacing the SoftMax Loss
• Single Layer Predictive Normalized Maximum Likelihood for Out-of-Distribution Detection
• On the Out-of-distribution Generalization of Probabilistic Image Modelling
• What Does Rotation Prediction Tell Us about Classifier Accuracy under Varying Testing Environments?
• Estimating Predictive Uncertainty Under Program Data Distribution Shift
• Unsupervised Anomaly Detection From Semantic Similarity Scores
• Shifting Transformation Learning for Out-of-Distribution Detection
• Modeling Discriminative Representations for Out-of-Domain Detection with Supervised Contrastive Learning
• Model-agnostic out-of-distribution detection using combined statistical tests
• Combating the effects of speed and delays in end-to-end self-driving
• Exploring Vicinal Risk Minimization for Lightweight Out-of-Distribution Detection
• Rainproof: An Umbrella To Shield Text Generators From Out-Of-Distribution Data
• Bridging In- and Out-of-distribution Samples for Their Better Discriminability
• Identifying Untrustworthy Predictions in Neural Networks by Geometric Gradient Analysis
• Attack as Defense: Characterizing Adversarial Examples using Robustness
• A Distance-based Anomaly Detection Framework for Deep Reinforcement Learning
• Classification and Uncertainty Quantification of Corrupted Data using Semi-Supervised Autoencoders
• An Infinite-Feature Extension for Bayesian ReLU Nets That Fixes Their Asymptotic Overconfidence
• Propagated Perturbation of Adversarial Attack for well-known CNNs: Empirical Study and its Explanation
• Likelihood Ratios and Generative Classifiers for Unsupervised Out-of-Domain Detection In Task Oriented Dialog
• Semi-Supervised Class Discovery
• Adversarial TCAV -- Robust and Effective Interpretation of Intermediate Layers in Neural Networks
• Brittle Features May Help Anomaly Detection
• Learning to Cascade: Confidence Calibration for Improving the Accuracy and Computational Cost of Cascade Inference Systems
• What do CNN neurons learn: Visualization & Clustering
• Being Single Has Benefits. Instance Poisoning to Deceive Malware Classifiers
• MOOD: Multi-level Out-of-distribution Detection
• Energy-Based Anomaly Detection and Localization
• A Survey on Assessing the Generalization Envelope of Deep Neural Networks: Predictive Uncertainty, Out-of-distribution and Adversarial Samples
• Test Sample Accuracy Scales with Training Sample Density in Neural Networks
• Improved Robustness to Open Set Inputs via Tempered Mixup
• Robust Deep Learning Ensemble against Deception
• Improving Calibration and Out-of-Distribution Detection in Medical Image Segmentation with Convolutional Neural Networks
• Adversarial Examples Detection beyond Image Space
• Decomposing Representations for Deterministic Uncertainty Estimation
• Interpretable Disentanglement of Neural Networks by Extracting Class-Specific Subnetwork
• Beating Attackers At Their Own Games: Adversarial Example Detection Using Adversarial Gradient Directions
• Out-distribution aware Self-training in an Open World Setting
• Deep Residual Flow for Out of Distribution Detection
• Risk Management Framework for Machine Learning Security
• Unsupervised Out-of-Distribution Dialect Detection with Mahalanobis Distance
• Deep Out-of-Distribution Uncertainty Quantification via Weight Entropy Maximization
• Adversarial Attacks on ML Defense Models Competition
• Uncertainty quantification for White Matter Hyperintensity segmentation detects silent failures and improves automated Fazekas quantification
• Beyond Confidence: Adaptive Abstention in Dual-Threshold Conformal Prediction for Autonomous System Perception
• Out-Of-Distribution Detection With Subspace Techniques And Probabilistic Modeling Of Features
• Robust Anomaly Detection for Particle Physics Using Multi-Background Representation Learning
• A Multi-Scale Framework for Out-of-Distribution Detection in Dermoscopic Images
• Robust Out-of-Distribution Detection on Deep Probabilistic Generative Models
• Relevant-features based Auxiliary Cells for Energy Efficient Detection of Natural Errors
• DOI: Divergence-based Out-of-Distribution Indicators via Deep Generative Models
• A statistical framework for efficient out of distribution detection in deep neural networks
• A Hierarchical Feature Constraint to Camouflage Medical Adversarial Attacks
• The Compact Support Neural Network
• BOSH: An Efficient Meta Algorithm for Decision-based Attacks
• Out of distribution detection for skin and malaria images
• Kullback-Leibler Divergence-Based Out-of-Distribution Detection with Flow-Based Generative Models
• Natural Attribute-based Shift Detection
• BED: Bi-Encoder-Based Detectors for Out-of-Distribution Detection
• Generalizing Neural Networks by Reflecting Deviating Data in Production
• Plug-and-Play Anomaly Detection with Expectation Maximization Filtering
• Generalized One-Class Learning Using Pairs of Complementary Classifiers
• The Effect of Optimization Methods on the Robustness of Out-of-Distribution Detection Approaches
• Out-of-Distribution Detection for Dermoscopic Image Classification
• Learning To Characterize Adversarial Subspaces
• Understanding the Role of Self-Supervised Learning in Out-of-Distribution Detection Task
• Why Should we Combine Training and Post-Training Methods for Out-of-Distribution Detection?
• Neural Bootstrapper
• Adversarial Machine Learning for Cybersecurity and Computer Vision: Current Developments and Challenges
• Deep Learning-Based Autonomous Driving Systems: A Survey of Attacks and Defenses
• Attack-agnostic Adversarial Detection on Medical Data Using Explainable Machine Learning
• Probabilistic Trust Intervals for Out of Distribution Detection
• Interpretable BoW Networks for Adversarial Example Detection
• Out-of-Distribution Example Detection in Deep Neural Networks using Distance to Modelled Embedding
• Likelihood Landscapes: A Unifying Principle Behind Many Adversarial Defenses
• Continual Novelty Detection
• Semi-Supervised Text Classification via Self-Pretraining
• Learning to Separate Clusters of Adversarial Representations for Robust Adversarial Detection
• Joint Distribution across Representation Space for Out-of-Distribution Detection
• State-Reification Networks: Improving Generalization by Modeling the Distribution of Hidden Representations
• Revealing the Distributional Vulnerability of Discriminators by Implicit Generators
• Information Theoretic Structured Generative Modeling
• Meta Learning Low Rank Covariance Factors for Energy-Based Deterministic Uncertainty
• Combining Diverse Feature Priors
• On the detection of Out-Of-Distribution samples in Multiple Instance Learning
• Multi-Class Anomaly Detection
• Understanding Classifier Mistakes with Generative Models
• FROB: Few-shot ROBust Model for Classification and Out-of-Distribution Detection
• Why Out-of-distribution Detection in CNNs Does Not Like Mahalanobis -- and What to Use Instead
• Improving robustness of classifiers by training against live traffic
• Accumulative Poisoning Attacks on Real-time Data
• Model Rectification via Unknown Unknowns Extraction from Deployment Samples
• Improving Compositionality of Neural Networks by Decoding Representations to Inputs