The Differential Entropy of Mixtures: New Bounds and Applications

Abstract

Mixture distributions are extensively used as a modeling tool in diverse areas from machine learning to communications engineering to physics, and obtaining bounds on the entropy of probability distributions is of fundamental importance in many of these applications. This article provides sharp bounds on the entropy concavity deficit, which is the difference between the entropy of the mixture and the weighted sum of entropies of constituent components. Toward establishing lower and upper bounds on the concavity deficit, results that are of importance in their own right are obtained. In order to obtain nontrivial upper bounds, properties of the skew-divergence are developed and notions of "skew" $f$-divergences are introduced; a reverse Pinsker inequality and a bound on Jensen-Shannon divergence are obtained along the way. Complementary lower bounds are derived with special attention paid to the case that corresponds to independent summation of a continuous and a discrete random variable. Several applications of the bounds are delineated, including to mutual information of additive noise channels, thermodynamics of computation, and functional inequalities.

References in corpus (29)

• Entropy production along a stochastic trajectory and an integral fluctuation theorem
• Equilibrium free energy differences from nonequilibrium measurements: a master equation approach
• $f$-divergence Inequalities
• Properties of Classical and Quantum Jensen-Shannon Divergence
• Refined Second Law of Thermodynamics for fast random processes
• On a generalization of the Jensen-Shannon divergence
• Steady state statistics of driven diffusions
• Fluctuation relations in simple examples of non-equilibrium steady states
• The entropy per coordinate of a random vector is highly constrained under convexity conditions
• Beyond the entropy power inequality, via rearrangements
• Forward and Reverse Entropy Power Inequalities in Convex Geometry
• Concentration of the information in data with log-concave distributions
• Quantum Skew Divergence
• Optimal Concentration of Information Content For Log-Concave Densities
• Entropy bounds on abelian groups and the Ruzsa divergence
• On $f$-Divergences: Integral Representations, Local Behavior, and Inequalities
• Measure concentration and the weak Pinsker property
• Combinatorial Entropy Power Inequalities: A Preliminary Study of the Stam region
• Entropy Inequalities for Sums in Prime Cyclic Groups
• On Relations Between the Relative entropy and ${\chi }^2$-Divergence, Generalizations and Applications
• Majorization and Rényi Entropy Inequalities via Sperner Theory
• A series of maximum entropy upper bounds of the differential entropy
• Memory Erasure using Time Multiplexed Potentials
• Two remarks on generalized entropy power inequalities
• Concentration of information content for convex measures
• Wasserstein Stability of the Entropy Power Inequality for Log-Concave Densities
• Stability of the Shannon-Stam inequality via the Föllmer process
• Error Bounds on a Mixed Entropy Inequality
• Strongly Convex Divergences

Cited by in corpus (2)

• On Relations Between the Relative entropy and ${\chi }^2$-Divergence, Generalizations and Applications
• Realizing Information Erasure in Finite Time