Jensen-Shannon Divergence Based Novel Loss Functions for Bayesian Neural Networks

Abstract

Bayesian neural networks (BNNs) are state-of-the-art machine learning methods that can naturally regularize and systematically quantify uncertainties using their stochastic parameters. Kullback-Leibler (KL) divergence-based variational inference used in BNNs suffers from unstable optimization and challenges in approximating light-tailed posteriors due to the unbounded nature of the KL divergence. To resolve these issues, we formulate a novel loss function for BNNs based on a new modification to the generalized Jensen-Shannon (JS) divergence, which is bounded. In addition, we propose a Geometric JS divergence-based loss, which is computationally efficient since it can be evaluated analytically. We found that the JS divergence-based variational inference is intractable, and hence employed a constrained optimization framework to formulate these losses. Our theoretical analysis and empirical experiments on multiple regression and classification data sets suggest that the proposed losses perform better than the KL divergence-based loss, especially when the data sets are noisy or biased. Specifically, there are approximately 5% and 8% improvements in accuracy for a noise-added CIFAR-10 dataset and a regression dataset, respectively. There is about a 13% reduction in false negative predictions of a biased histopathology dataset. In addition, we quantify and compare the uncertainty metrics for the regression and classification tasks.

Accepted for publication in Neurocomputing

References in corpus (8)

• Proceedings of the 29th International Conference on Machine Learning (ICML-12)
• Hands-on Bayesian Neural Networks -- a Tutorial for Deep Learning Users
• Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks
• Bayesian Neural Networks: An Introduction and Survey
• Nuclear charge radii in Bayesian neural networks revisited
• Robust Simulation-Based Inference in Cosmology with Bayesian Neural Networks
• f-Divergence Variational Inference
• Constraining Variational Inference with Geometric Jensen-Shannon Divergence