Generalized Bhattacharyya and Chernoff upper bounds on Bayes error using quasi-arithmetic means

Abstract

Bayesian classification labels observations based on given prior information, namely class-a priori and class-conditional probabilities. Bayes' risk is the minimum expected classification cost that is achieved by the Bayes' test, the optimal decision rule. When no cost incurs for correct classification and unit cost is charged for misclassification, Bayes' test reduces to the maximum a posteriori decision rule, and Bayes risk simplifies to Bayes' error, the probability of error. Since calculating this probability of error is often intractable, several techniques have been devised to bound it with closed-form formula, introducing thereby measures of similarity and divergence between distributions like the Bhattacharyya coefficient and its associated Bhattacharyya distance. The Bhattacharyya upper bound can further be tightened using the Chernoff information that relies on the notion of best error exponent. In this paper, we first express Bayes' risk using the total variation distance on scaled distributions. We then elucidate and extend the Bhattacharyya and the Chernoff upper bound mechanisms using generalized weighted means. We provide as a byproduct novel notions of statistical divergences and affinity coefficients. We illustrate our technique by deriving new upper bounds for the univariate Cauchy and the multivariate $t$-distributions, and show experimentally that those bounds are not too distant to the computationally intractable Bayes' error.

22 pages, include R code. To appear in Pattern Recognition Letters

Cited by in corpus (14)

• On a generalization of the Jensen-Shannon divergence and the JS-symmetrization of distances relying on abstract means
• An elementary introduction to information geometry
• Estimating Mixture Entropy with Pairwise Distances
• On a Variational Definition for the Jensen-Shannon Symmetrization of Distances based on the Information Radius
• Revisiting Chernoff Information with Likelihood Ratio Exponential Families
• Proof-of-work consensus by quantum sampling
• Evaluating State-of-the-Art Classification Models Against Bayes Optimality
• Strongly Convex Divergences
• Generalizing Jensen and Bregman divergences with comparative convexity and the statistical Bhattacharyya distances with comparable means
• Quantification of mismatch error in randomly switching linear state-space models
• $\alpha$-Geodesical Skew Divergence
• The α-divergences associated with a pair of strictly comparable quasi-arithmetic means
• On The Chain Rule Optimal Transport Distance
• Two-dimensional Bhattacharyya bound linear discriminant analysis with its applications