$f$-divergence Inequalities

Abstract

This paper develops systematic approaches to obtain $f$-divergence inequalities, dealing with pairs of probability measures defined on arbitrary alphabets. Functional domination is one such approach, where special emphasis is placed on finding the best possible constant upper bounding a ratio of $f$-divergences. Another approach used for the derivation of bounds among $f$-divergences relies on moment inequalities and the logarithmic-convexity property, which results in tight bounds on the relative entropy and Bhattacharyya distance in terms of ${\chi }^2$ divergences. A rich variety of bounds are shown to hold under boundedness assumptions on the relative information. Special attention is devoted to the total variation distance and its relation to the relative information and relative entropy, including "reverse Pinsker inequalities," as well as on the $E_{\gamma }$ divergence, which generalizes the total variation distance. Pinsker's inequality is extended for this type of $f$-divergence, a result which leads to an inequality linking the relative entropy and relative information spectrum. Integral expressions of the Rényi divergence in terms of the relative information spectrum are derived, leading to bounds on the Rényi divergence in terms of either the variational distance or relative entropy.

IEEE Trans. on Information Theory, vol. 62, no. 11, pp. 5973--6006, November 2016. This manuscript is identical to the journal paper, apart of some additional material which includes Sections III-C and IV-F, and three technical proofs

References in corpus (16)

• Rényi Divergence and Kullback-Leibler Divergence
• $f$-divergence Inequalities
• A Tight Upper Bound for the Third-Order Asymptotics for Most Discrete Memoryless Channels
• Continuity bounds on the quantum relative entropy
• On Pinsker's Type Inequalities and Csiszar's f-divergences. Part I: Second and Fourth-Order Inequalities
• Continuity bounds on the quantum relative entropy - II
• Optimal Quantization for Distribution Synthesis
• Entropy Bounds for Discrete Random Variables via Maximal Coupling
• On Bayes Risk Lower Bounds
• On the asymmetry of the relative entropy
• Tight Bounds for Symmetric Divergence Measures and a Refined Bound for Lossless Source Coding
• Strong Converse Theorems for Classes of Multimessage Multicast Networks: A Rényi Divergence Approach
• Resolvability in Eγ with Applications to Lossy Compression and Wiretap Channels
• Concentration of Measure Inequalities and Their Communication and Information-Theoretic Applications
• $E_{\gamma }$-Resolvability
• Refinement of some moment inequalities

Cited by in corpus (75)

• $f$-divergence Inequalities
• On a generalization of the Jensen-Shannon divergence
• Estimating Mixture Entropy with Pairwise Distances
• Fast quantum circuit cutting with randomized measurements
• Counterfactual Sensitivity and Robustness
• Optimal Quantization for Distribution Synthesis
• On $f$-Divergences: Integral Representations, Local Behavior, and Inequalities
• Limits of Deepfake Detection: A Robust Estimation Viewpoint
• The Discrete Gaussian for Differential Privacy
• The Renyi Capacity and Center
• On explicit $L^2$-convergence rate estimate for underdamped Langevin dynamics
• Quantum Rényi and $f$-divergences from integral representations
• On Relations Between the Relative entropy and ${\chi }^2$-Divergence, Generalizations and Applications
• On Data-Processing and Majorization Inequalities for $f$-Divergences with Applications
• Maxwell's Demon walks into Wall Street: Stochastic Thermodynamics meets Expected Utility Theory
• Two Measures of Dependence
• The Differential Entropy of Mixtures: New Bounds and Applications
• Integral formula for quantum relative entropy implies data processing inequality
• Generalization Error Bounds Via Rényi-, $f$-Divergences and Maximal Leakage
• Geometric approach to quantum statistical inference
• Accelerating Bayesian microseismic event location with deep learning
• Thermodynamic correlation inequality
• f-Divergence Variational Inference
• Variational regularisation for inverse problems with imperfect forward operators and general noise models
• The Sphere Packing Bound via Augustin's Method
• Linear Bounds between Contraction Coefficients for $f$-Divergences
• A Note on Reverse Pinsker Inequalities
• Quantum chi-squared tomography and mutual information testing
• Methods for Quantifying Dataset Similarity: a Review, Taxonomy and Comparison
• Bottleneck Problems: Information and Estimation-Theoretic View
• Proximity Operators of Discrete Information Divergences
• On $f$-divergences between Cauchy distributions
• Effective Sample Size, Dimensionality, and Generalization in Covariate Shift Adaptation
• Non-saturating GAN training as divergence minimization
• Entropic Regularization of Markov Decision Processes
• To Split or Not to Split: The Impact of Disparate Treatment in Classification
• Data-Driven Estimation of Capacity Upper Bounds
• Higher-order propagation of chaos in $L^2$ for interacting diffusions
• Information thermodynamics: from physics to neuroscience
• Properties of a Generalized Divergence Related to Tsallis Relative Entropy
• Obfuscation via Information Density Estimation
• Strongly Convex Divergences
• Quantum $f$-divergences via Nussbaum-Szkoła Distributions and Applications to $f$-divergence Inequalities
• A strong converse bound for multiple hypothesis testing, with applications to high-dimensional estimation
• On the All-Or-Nothing Behavior of Bernoulli Group Testing
• An Optimal Transport View on Generalization
• Lower Bounds for Learning Quantum States with Single-Copy Measurements
• Bias Correction with Jackknife, Bootstrap, and Taylor Series
• Continuity of Generalized Entropy and Statistical Learning
• Computing Differential Privacy Guarantees for Heterogeneous Compositions Using FFT
• Hessian-Free High-Resolution Nesterov Acceleration for Sampling
• Simple Majority Consensus in Networks with Unreliable Communication
• Bregman Divergence Bounds and Universality Properties of the Logarithmic Loss
• Systematic approaches to generate reversiblizations of Markov chains
• An Information-theoretic Approach to Distribution Shifts
• Dual Loomis-Whitney inequalities via information theory
• GANs with Conditional Independence Graphs: On Subadditivity of Probability Divergences
• Hypothesis Testing under Mutual Information Privacy Constraints in the High Privacy Regime
• Semiparametric two-component mixture models under L-moments constraints
• Semiparametric two-component mixture models under linear constraints
• Bounds on the rates of statistical divergences and mutual information via stochastic thermodynamics
• Wasserstein Introspective Neural Networks
• Neural Computation of Capacity Region of Memoryless Multiple Access Channels
• Bounds on Covert Capacity with Sub-Exponential Random Slot Selection
• Constrained Instance and Class Reweighting for Robust Learning under Label Noise
• Reverse-type Data Processing Inequality
• Non-Markovian superposition process model for stochastically describing concentration-discharge relationship
• Online learning of a panoply of quantum objects
• Estimating Free Parameters in Stochastic Oscillatory Models Using a Weighted Cost Function
• An Information Theoretic Treatment of Yager's Probability Distribution Negation
• Cautious Policy Programming: Exploiting KL Regularization in Monotonic Policy Improvement for Reinforcement Learning
• Two-Moment Inequalities for Rényi Entropy and Mutual Information
• On the $\Phi$-Stability and Related Conjectures
• Testing identity of collections of quantum states: sample complexity analysis
• CNN with large memory layers