$f$-divergence Inequalities

Abstract

This paper develops systematic approaches to obtain $f$-divergence inequalities, dealing with pairs of probability measures defined on arbitrary alphabets. Functional domination is one such approach, where special emphasis is placed on finding the best possible constant upper bounding a ratio of $f$-divergences. Another approach used for the derivation of bounds among $f$-divergences relies on moment inequalities and the logarithmic-convexity property, which results in tight bounds on the relative entropy and Bhattacharyya distance in terms of ${\chi }^2$ divergences. A rich variety of bounds are shown to hold under boundedness assumptions on the relative information. Special attention is devoted to the total variation distance and its relation to the relative information and relative entropy, including "reverse Pinsker inequalities," as well as on the $E_{\gamma }$ divergence, which generalizes the total variation distance. Pinsker's inequality is extended for this type of $f$-divergence, a result which leads to an inequality linking the relative entropy and relative information spectrum. Integral expressions of the Rényi divergence in terms of the relative information spectrum are derived, leading to bounds on the Rényi divergence in terms of either the variational distance or relative entropy.

IEEE Trans. on Information Theory, vol. 62, no. 11, pp. 5973--6006, November 2016. This manuscript is identical to the journal paper, apart of some additional material which includes Sections III-C and IV-F, and three technical proofs

References in corpus (5)

• $f$-divergence Inequalities
• Resolvability in Eγ with Applications to Lossy Compression and Wiretap Channels
• Concentration of Measure Inequalities and Their Communication and Information-Theoretic Applications
• $E_{\gamma }$-Resolvability
• Refinement of some moment inequalities

Cited by in corpus (74)

• $f$-divergence Inequalities
• On a generalization of the Jensen-Shannon divergence
• Estimating Mixture Entropy with Pairwise Distances
• Fast quantum circuit cutting with randomized measurements
• Counterfactual Sensitivity and Robustness
• On $f$-Divergences: Integral Representations, Local Behavior, and Inequalities
• Limits of Deepfake Detection: A Robust Estimation Viewpoint
• The Discrete Gaussian for Differential Privacy
• The Renyi Capacity and Center
• On explicit $L^2$-convergence rate estimate for underdamped Langevin dynamics
• Quantum Rényi and $f$-divergences from integral representations
• On Relations Between the Relative entropy and ${\chi }^2$-Divergence, Generalizations and Applications
• On Data-Processing and Majorization Inequalities for $f$-Divergences with Applications
• Maxwell's Demon walks into Wall Street: Stochastic Thermodynamics meets Expected Utility Theory
• Two Measures of Dependence
• The Differential Entropy of Mixtures: New Bounds and Applications
• Integral formula for quantum relative entropy implies data processing inequality
• Geometric approach to quantum statistical inference
• Generalization Error Bounds Via Rényi-, $f$-Divergences and Maximal Leakage
• Accelerating Bayesian microseismic event location with deep learning
• Thermodynamic correlation inequality
• f-Divergence Variational Inference
• The Sphere Packing Bound via Augustin's Method
• Variational regularisation for inverse problems with imperfect forward operators and general noise models
• A Note on Reverse Pinsker Inequalities
• Linear Bounds between Contraction Coefficients for $f$-Divergences
• Methods for Quantifying Dataset Similarity: a Review, Taxonomy and Comparison
• Quantum chi-squared tomography and mutual information testing
• Proximity Operators of Discrete Information Divergences
• On $f$-divergences between Cauchy distributions
• Bottleneck Problems: Information and Estimation-Theoretic View
• Effective Sample Size, Dimensionality, and Generalization in Covariate Shift Adaptation
• Entropic Regularization of Markov Decision Processes
• Non-saturating GAN training as divergence minimization
• Data-Driven Estimation of Capacity Upper Bounds
• Higher-order propagation of chaos in $L^2$ for interacting diffusions
• Information thermodynamics: from physics to neuroscience
• To Split or Not to Split: The Impact of Disparate Treatment in Classification
• Properties of a Generalized Divergence Related to Tsallis Relative Entropy
• Obfuscation via Information Density Estimation
• Strongly Convex Divergences
• A strong converse bound for multiple hypothesis testing, with applications to high-dimensional estimation
• Quantum $f$-divergences via Nussbaum-Szkoła Distributions and Applications to $f$-divergence Inequalities
• Lower Bounds for Learning Quantum States with Single-Copy Measurements
• An Optimal Transport View on Generalization
• On the All-Or-Nothing Behavior of Bernoulli Group Testing
• Bias Correction with Jackknife, Bootstrap, and Taylor Series
• Continuity of Generalized Entropy and Statistical Learning
• Hessian-Free High-Resolution Nesterov Acceleration for Sampling
• Computing Differential Privacy Guarantees for Heterogeneous Compositions Using FFT
• Simple Majority Consensus in Networks with Unreliable Communication
• Bregman Divergence Bounds and Universality Properties of the Logarithmic Loss
• Systematic approaches to generate reversiblizations of Markov chains
• An Information-theoretic Approach to Distribution Shifts
• Dual Loomis-Whitney inequalities via information theory
• Hypothesis Testing under Mutual Information Privacy Constraints in the High Privacy Regime
• GANs with Conditional Independence Graphs: On Subadditivity of Probability Divergences
• Semiparametric two-component mixture models under linear constraints
• Semiparametric two-component mixture models under L-moments constraints
• Constrained Instance and Class Reweighting for Robust Learning under Label Noise
• Bounds on the rates of statistical divergences and mutual information via stochastic thermodynamics
• Wasserstein Introspective Neural Networks
• Bounds on Covert Capacity with Sub-Exponential Random Slot Selection
• Reverse-type Data Processing Inequality
• Non-Markovian superposition process model for stochastically describing concentration-discharge relationship
• Neural Computation of Capacity Region of Memoryless Multiple Access Channels
• On the $\Phi$-Stability and Related Conjectures
• Cautious Policy Programming: Exploiting KL Regularization in Monotonic Policy Improvement for Reinforcement Learning
• Estimating Free Parameters in Stochastic Oscillatory Models Using a Weighted Cost Function
• An Information Theoretic Treatment of Yager's Probability Distribution Negation
• Testing identity of collections of quantum states: sample complexity analysis
• CNN with large memory layers
• Two-Moment Inequalities for Rényi Entropy and Mutual Information
• Online learning of a panoply of quantum objects