Prior Activation Distribution (PAD): A Versatile Representation to Utilize DNN Hidden Units
arXiv:1907.02711
Abstract
In this paper, we introduce the concept of Prior Activation Distribution (PAD) as a versatile and general technique to capture the typical activation patterns of hidden layer units of a Deep Neural Network used for classification tasks. We show that the combined neural activations of such a hidden layer have class-specific distributional properties, and then define multiple statistical measures to compute how far a test sample's activations deviate from such distributions. Using a variety of benchmark datasets (including MNIST, CIFAR10, Fashion-MNIST & notMNIST), we show how such PAD-based measures can be used, independent of any training technique, to (a) derive fine-grained uncertainty estimates for inferences; (b) provide inferencing accuracy competitive with alternatives that require execution of the full pipeline, and (c) reliably isolate out-of-distribution test samples.
Submitted to NeurIPS 2019
References in corpus (8)
- A Survey on Deep Learning in Medical Image Analysis
- Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
- Pruning Convolutional Neural Networks for Resource Efficient Inference
- Bayesian Convolutional Neural Networks with Bernoulli Approximate Variational Inference
- Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks
- Evidential Deep Learning to Quantify Classification Uncertainty
- Understanding intermediate layers using linear classifier probes
- Understanding the Disharmony between Dropout and Batch Normalization by Variance Shift