Outside the Box: Abstraction-Based Monitoring of Neural Networks

Abstract

Neural networks have demonstrated unmatched performance in a range of classification tasks. Despite numerous efforts of the research community, novelty detection remains one of the significant limitations of neural networks. The ability to identify previously unseen inputs as novel is crucial for our understanding of the decisions made by neural networks. At runtime, inputs not falling into any of the categories learned during training cannot be classified correctly by the neural network. Existing approaches treat the neural network as a black box and try to detect novel inputs based on the confidence of the output predictions. However, neural networks are not trained to reduce their confidence for novel inputs, which limits the effectiveness of these approaches. We propose a framework to monitor a neural network by observing the hidden layers. We employ a common abstraction from program analysis - boxes - to identify novel behaviors in the monitored layers, i.e., inputs that cause behaviors outside the box. For each neuron, the boxes range over the values seen in training. The framework is efficient and flexible to achieve a desired trade-off between raising false warnings and detecting novel inputs. We illustrate the performance and the robustness to variability in the unknown classes on popular image-classification benchmarks.

arXiv commentaccepted at ECAI 2020

References in corpus (18)

• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
• Towards A Rigorous Science of Interpretable Machine Learning
• End to End Learning for Self-Driving Cars
• Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning
• Unsupervised Domain Adaptation by Backpropagation
• XLNet: Generalized Autoregressive Pretraining for Language Understanding
• On Calibration of Modern Neural Networks
• A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks
• Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles
• Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks
• To Trust Or Not To Trust A Classifier
• Selective Classification for Deep Neural Networks
• Combating Label Noise in Deep Learning Using Abstention
• Fixing the train-test resolution discrepancy
• Online k-means Clustering
• Runtime Monitoring Neuron Activation Patterns
• KS(conf ): A Light-Weight Test if a ConvNet Operates Outside of Its Specifications

Cited by in corpus (5)

• Run-Time Monitoring of Machine Learning for Robotic Perception: A Survey of Emerging Trends
• Into the Unknown: Active Monitoring of Neural Networks
• Model Assertions for Monitoring and Improving ML Models
• Input Validation for Neural Networks via Runtime Local Robustness Verification
• SpecRepair: Counter-Example Guided Safety Repair of Deep Neural Networks