DS-UI: Dual-Supervised Mixture of Gaussian Mixture Models for Uncertainty Inference

Abstract

This paper proposes a dual-supervised uncertainty inference (DS-UI) framework for improving Bayesian estimation-based uncertainty inference (UI) in deep neural network (DNN)-based image recognition. In the DS-UI, we combine the classifier of a DNN, i.e., the last fully-connected (FC) layer, with a mixture of Gaussian mixture models (MoGMM) to obtain an MoGMM-FC layer. Unlike existing UI methods for DNNs, which only calculate the means or modes of the DNN outputs' distributions, the proposed MoGMM-FC layer acts as a probabilistic interpreter for the features that are inputs of the classifier to directly calculate the probability density of them for the DS-UI. In addition, we propose a dual-supervised stochastic gradient-based variational Bayes (DS-SGVB) algorithm for the MoGMM-FC layer optimization. Unlike conventional SGVB and optimization algorithms in other UI methods, the DS-SGVB not only models the samples in the specific class for each Gaussian mixture model (GMM) in the MoGMM, but also considers the negative samples from other classes for the GMM to reduce the intra-class distances and enlarge the inter-class margins simultaneously for enhancing the learning ability of the MoGMM-FC layer in the DS-UI. Experimental results show the DS-UI outperforms the state-of-the-art UI methods in misclassification detection. We further evaluate the DS-UI in open-set out-of-domain/-distribution detection and find statistically significant improvements. Visualizations of the feature spaces demonstrate the superiority of the DS-UI.

References in corpus (22)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Explaining and Harnessing Adversarial Examples
• Improving neural networks by preventing co-adaptation of feature detectors
• Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning
• On Calibration of Modern Neural Networks
• A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks
• Weight Uncertainty in Neural Networks
• A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks
• The Devil is in the Channels: Mutual-Channel Loss for Fine-Grained Image Classification
• Predictive Uncertainty Estimation via Prior Networks
• Uncertainty Estimation Using a Single Deep Deterministic Neural Network
• Weakly Supervised Attention Pyramid Convolutional Neural Network for Fine-Grained Visual Classification
• Bayesian Uncertainty Estimation for Batch Normalized Deep Networks
• Preconditioned Stochastic Gradient Langevin Dynamics for Deep Neural Networks
• Learned Image Downscaling for Upscaling using Content Adaptive Resampler
• Reverse KL-Divergence Training of Prior Networks: Improved Uncertainty and Adversarial Robustness
• Identifying Mixtures of Mixtures Using Bayesian Estimation
• SDE-Net: Equipping Deep Neural Networks with Uncertainty Estimates
• Getting a CLUE: A Method for Explaining Uncertainty Estimates
• Inhibited Softmax for Uncertainty Estimation in Neural Networks
• Fast Second-Order Stochastic Backpropagation for Variational Inference