Rényi Divergence and Kullback-Leibler Divergence

Abstract

Rényi divergence is related to Rényi entropy much like Kullback-Leibler divergence is related to Shannon's entropy, and comes up in many settings. It was introduced by Rényi as a measure of information that satisfies almost the same axioms as Kullback-Leibler divergence, and depends on a parameter that is called its order. In particular, the Rényi divergence of order 1 equals the Kullback-Leibler divergence. We review and extend the most important properties of Rényi divergence and Kullback-Leibler divergence, including convexity, continuity, limits of $\sigma$-algebras and the relation of the special order 0 to the Gaussian dichotomy and contiguity. We also show how to generalize the Pythagorean inequality to orders different from 1, and we extend the known equivalence between channel capacity and minimax redundancy to continuous channel inputs (for all orders) and present several other minimax results.

To appear in IEEE Transactions on Information Theory

References in corpus (7)

• From $\epsilon$-entropy to KL-entropy: Analysis of minimum information complexity density estimation
• On Pinsker's Type Inequalities and Csiszar's f-divergences. Part I: Second and Fourth-Order Inequalities
• Multiple Source Adaptation and the Renyi Divergence
• Estimators, escort probabilities, and phi-exponential families in statistical physics
• Interpretations of Renyi Entropies And Divergences
• Rényi Divergence and Majorization
• A Note on a Characterization of Rényi Measures and its Relation to Composite Hypothesis Testing

Cited by in corpus (310)

• Renyi Differential Privacy
• The Perception-Distortion Tradeoff
• A review of domain adaptation without target labels
• Scalable Private Learning with PATE
• $f$-divergence Inequalities
• An introduction to domain adaptation and transfer learning
• Federated Learning with Bayesian Differential Privacy
• Differentially Private Model Publishing for Deep Learning
• On a generalization of the Jensen-Shannon divergence
• Fusion of Probability Density Functions
• Privacy Amplification by Iteration
• Dropout Inference in Bayesian Neural Networks with Alpha-divergences
• Information fractal dimension of mass function
• ordpy: A Python package for data analysis with permutation entropy and ordinal network methods
• Correlating thermal machines and the second law at the nanoscale
• Regimes of classical simulability for noisy Gaussian boson sampling
• Rényi Differential Privacy of the Sampled Gaussian Mechanism
• Estimating Mixture Entropy with Pairwise Distances
• On Variational Expressions for Quantum Relative Entropies
• Community detection in multi-relational data with restricted multi-layer stochastic blockmodel
• Stochastic independence as a resource in small-scale thermodynamics
• Beyond the entropy power inequality, via rearrangements
• Spectral MLE: Top-$K$ Rank Aggregation from Pairwise Comparisons
• Concentrated Differential Privacy: Simplifications, Extensions, and Lower Bounds
• Rethinking Lossy Compression: The Rate-Distortion-Perception Tradeoff
• Generalized Variational Inference: Three arguments for deriving new Posteriors
• Operational Interpretation of Renyi Information Measures via Composite Hypothesis Testing Against Product and Markov Distributions
• Guaranteed bounds on the Kullback-Leibler divergence of univariate mixtures using piecewise log-sum-exp inequalities
• Variational Inference of Disentangled Latent Concepts from Unlabeled Observations
• Optimal Quantization for Distribution Synthesis
• Consistency of Variational Bayes Inference for Estimation and Model Selection in Mixtures
• Generalization Bounds via Information Density and Conditional Information Density
• Generating Synthetic Mobility Networks with Generative Adversarial Networks
• Uniqueness and characterization theorems for generalized entropies
• On $f$-Divergences: Integral Representations, Local Behavior, and Inequalities
• A generalization of majorization that characterizes Shannon entropy
• Entropic Steering Criteria: Applications to Bipartite and Tripartite Systems
• Semantic Security via Seeded Modular Coding Schemes and Ramanujan Graphs
• OutbreakFlow: Model-based Bayesian inference of disease outbreak dynamics with invertible neural networks and its application to the COVID-19 pandemics in Germany
• Large-Scale Methods for Distributionally Robust Optimization
• Robust Hypothesis Testing with $\alpha$-Divergence
• Formal Groups and $Z$-Entropies
• Characterization of Time Series Via Rényi Complexity-Entropy Curves
• Entropy and relative entropy from information-theoretic principles
• On a Variational Definition for the Jensen-Shannon Symmetrization of Distances based on the Information Radius
• Information-theoretic bounds for exact recovery in weighted stochastic block models using the Renyi divergence
• Exact solution for the quantum and private capacities of bosonic dephasing channels
• Lower Bounds for Locally Private Estimation via Communication Complexity
• The Renyi Capacity and Center
• Renyi relative entropies of quantum Gaussian states
• Bounding generalized relative entropies: Nonasymptotic quantum speed limits
• GS-WGAN: A Gradient-Sanitized Approach for Learning Differentially Private Generators
• Information Aware Max-Norm Dirichlet Networks for Predictive Uncertainty Estimation
• Defining quantum divergences via convex optimization
• Domain Adaptation and Autoencoder Based Unsupervised Speech Enhancement
• A Non-Asymptotic Analysis for Stein Variational Gradient Descent
• Pointwise Maximal Leakage
• Relative Entropy and Catalytic Relative Majorization
• On the Kullback-Leibler divergence between discrete normal distributions
• Decentralized and Model-Free Federated Learning: Consensus-Based Distillation in Function Space
• Approximate Span Liftings
• Scalable Information Inequalities for Uncertainty Quantification
• Differentially Private Federated Learning on Heterogeneous Data
• Gibbs posterior concentration rates under sub-exponential type losses
• Evaluating Model Robustness and Stability to Dataset Shift
• The Augustin Capacity and Center
• Coding for Positive Rate in the Source Model Key Agreement Problem
• Holographic second laws of black hole thermodynamics
• Quantum work statistics and resource theories: bridging the gap through Renyi divergences
• On The Classification-Distortion-Perception Tradeoff
• Subsampling Generative Adversarial Networks: Density Ratio Estimation in Feature Space with Softplus Loss
• Toward physical realizations of thermodynamic resource theories
• Contagion in financial systems: A Bayesian network approach
• Relations between Dissipated Work and Rényi Divergences
• Alpha-Beta Divergence For Variational Inference
• On Relations Between the Relative entropy and ${\chi }^2$-Divergence, Generalizations and Applications
• Analysis of Langevin Monte Carlo via convex optimization
• Revisiting Chernoff Information with Likelihood Ratio Exponential Families
• alpha-Deep Probabilistic Inference (alpha-DPI): efficient uncertainty quantification from exoplanet astrometry to black hole feature extraction
• Optimal Experimental Design Using A Consistent Bayesian Approach
• Improved bound on entropy production in a quantum annealer
• Properties of Noncommutative Renyi and Augustin Information
• Thermodynamic utility of Non-Markovianity from the perspective of resource interconversion
• Work fluctuation-dissipation trade-off in heat engines
• On Data-Processing and Majorization Inequalities for $f$-Divergences with Applications
• Internal entanglement and external correlations of any form limit each other
• DPIS: An Enhanced Mechanism for Differentially Private SGD with Importance Sampling
• Reasoning About Generalization via Conditional Mutual Information
• Maxwell's Demon walks into Wall Street: Stochastic Thermodynamics meets Expected Utility Theory
• Data-driven priors and their posterior concentration rates
• Quantifying non-Gaussianity of a quantum state by the negative entropy of quadrature distributions
• InfoAT: Improving Adversarial Training Using the Information Bottleneck Principle
• The variance of relative surprisal as single-shot quantifier
• A Joint Typicality Approach to Algebraic Network Information Theory
• Exponential decay of Rényi divergence under Fokker-Planck equations
• Riesz-Quincunx-UNet Variational Auto-Encoder for Satellite Image Denoising
• Pretty good quantum state transfer on isotropic and anisotropic Heisenberg spin chains with tailored site dependent exchange couplings
• Infomax strategies for an optimal balance between exploration and exploitation
• On-Average KL-Privacy and its equivalence to Generalization for Max-Entropy Mechanisms
• Autonomous Quantum Devices: When Are They Realizable without Additional Thermodynamic Costs?
• A Scale-invariant Generalization of the Rényi Entropy, Associated Divergences and their Optimizations under Tsallis' Nonextensive Framework
• The Sphere Packing Bound For Memoryless Channels
• Two Measures of Dependence
• Autofocused oracles for model-based design
• Information-Theoretic Perspectives on Brascamp-Lieb Inequality and Its Reverse
• Consistency of ELBO maximization for model selection
• Characterisation of quantum betting tasks in terms of Arimoto mutual information
• Amplifying Inter-message Distance: On Information Divergence Measures in Big Data
• Least $k$th-Order and Rényi Generative Adversarial Networks
• Contextual Distillation Model for Diversified Recommendation
• Importance of Small Probability Events in Big Data: Information Measures, Applications, and Challenges
• Generalization Error Bounds Via Rényi-, $f$-Divergences and Maximal Leakage
• Proximal Langevin Algorithm: Rapid Convergence Under Isoperimetry
• Refined Young inequality and its application to divergences
• Optimal Accounting of Differential Privacy via Characteristic Function
• Transport-majorization to analytic and geometric inequalities
• Mosaics of combinatorial designs for information-theoretic security
• Asymmetry and tighter uncertainty relations for Rényi entropies via quantum-classical decompositions of resource measures
• The Sphere Packing Bound via Augustin's Method
• On Reverse Pinsker Inequalities
• Refined Strong Converse for the Constant Composition Codes
• Scalable Variational Inference in Log-supermodular Models
• Infinite-dimensional gradient-based descent for alpha-divergence minimisation
• Methods for Quantifying Dataset Similarity: a Review, Taxonomy and Comparison
• The Reconfiguration Pattern of Individual Brain Metabolic Connectome for Parkinson's Disease Identification
• Quantum Foundations of Classical Reversible Computing
• Tight Bounds for Symmetric Divergence Measures and a Refined Bound for Lossless Source Coding
• Concurrent Composition Theorems for Differential Privacy
• DOCTOR: A Simple Method for Detecting Misclassification Errors
• Pufferfish Privacy: An Information-Theoretic Study
• Markov Categories and Entropy
• Convergence Rates of Variational Posterior Distributions
• Multiple Private Key Generation for Continuous Memoryless Sources with A Helper
• Variational f-divergence Minimization
• Duality between source coding with quantum side information and c-q channel coding
• Effective Sample Size, Dimensionality, and Generalization in Covariate Shift Adaptation
• Generalized Mixability via Entropic Duality
• Links between Dissipation and Rényi Divergences in $\mathcal{P}\mathcal{T}$-Symmetric Quantum Mechanics
• The MAC Resolvability Region, Semantic Security and Its Operational Implications
• An empirical $G$-Wishart prior for sparse high-dimensional Gaussian graphical models
• Leave Zero Out: Towards a No-Cross-Validation Approach for Model Selection
• DistFL: Distribution-aware Federated Learning for Mobile Scenarios
• Bayesian Differential Privacy for Machine Learning
• Maximum one-shot dissipated work from Renyi divergences
• Strong Converse Theorems for Classes of Multimessage Multicast Networks: A Rényi Divergence Approach
• Bayesian fractional posteriors
• Speed limits and thermodynamic uncertainty relations for quantum systems governed by non-Hermitian Hamiltonian
• On the Support Recovery of Jointly Sparse Gaussian Sources using Sparse Bayesian Learning
• Multi-Cause Deconfounding for Recommender Systems with Latent Confounders
• Point Information Gain and Multidimensional Data Analysis
• Quantum Channel Learning
• A Mathematical Theory of Attention
• The Effect of Quantization in Federated Learning: A Rényi Differential Privacy Perspective
• Relations between Heat Exchange and Rényi Divergences
• Relating relative Rényi entropies and Wigner-Yanase-Dyson skew information to generalized multiple quantum coherences
• A Simple Derivation of the Refined Sphere Packing Bound Under Certain Symmetry Hypotheses
• Renyi divergences from Euclidean quenches
• Operator-valued Schatten spaces and quantum entropies
• A New Approach to Adaptive Data Analysis and Learning via Maximal Leakage
• Differentially Private Confidence Intervals for Proportions under Stratified Random Sampling
• A Bounded Measure for Estimating the Benefit of Visualization
• Conditional Rényi entropy and the relationships between Rényi capacities
• A precise bare simulation approach to the minimization of some distances. Foundations
• A Bounded Measure for Estimating the Benefit of Visualization: Theoretical Discourse and Conceptual Evaluation
• Minimax Rényi Redundancy
• Estimation of KL Divergence: Optimal Minimax Rate
• Stochastic Proximal Langevin Algorithm: Potential Splitting and Nonasymptotic Rates
• Catalytic transformations for thermal operations
• Rationally Inattentive Inverse Reinforcement Learning Explains YouTube Commenting Behavior
• Dissipation in the Generalized Gibbs Ensemble
• Resolvability on Continuous Alphabets
• Strongly Convex Divergences
• Rényi divergence and the central limit theorem
• Complex-valued Federated Learning with Differential Privacy and MRI Applications
• Strong Brascamp-Lieb Inequalities
• On the Existence of the Augustin Mean
• Bounding the resources for thermalizing many-body localized systems
• Concentration properties of fractional posterior in 1-bit matrix completion
• Matching Users' Preference Under Target Revenue Constraints in Optimal Data Recommendation Systems
• Variational approximations using Fisher divergence
• Properties of a Generalized Divergence Related to Tsallis Relative Entropy
• Conditions for the existence of a generalization of Rényi divergence
• The Augustin Center and The Sphere Packing Bound For Memoryless Channels
• Hyperparameter Tuning with Renyi Differential Privacy
• Locally-Measured Rényi Divergences
• Generalizations of Fano's Inequality for Conditional Information Measures via Majorization Theory
• Concentration of a sparse Bayesian model with Horseshoe prior in estimating high-dimensional precision matrix
• Semi-Supervised Hypothesis Transfer for Source-Free Domain Adaptation
• Calibrating Noise to Variance in Adaptive Data Analysis
• High Resolution Face Editing with Masked GAN Latent Code Optimization
• Towards the Unification and Robustness of Perturbation and Gradient Based Explanations
• AED-PADA:Improving Generalizability of Adversarial Example Detection via Principal Adversarial Domain Adaptation
• Equivocations, Exponents and Second-Order Coding Rates under Various Rényi Information Measures
• Quantum $f$-divergences via Nussbaum-Szkoła Distributions and Applications to $f$-divergence Inequalities
• Always Lurking: Understanding and Mitigating Bias in Online Human Trafficking Detection
• Speed limit, dissipation bound and dissipation-time trade-off in thermal relaxation processes
• Point Divergence Gain and Multidimensional Data Sequences Analysis
• Storage Space Allocation Strategy for Digital Data with Message Importance
• Contextual Fine-to-Coarse Distillation for Coarse-grained Response Selection in Open-Domain Conversations
• Random Matrix Improved Covariance Estimation for a Large Class of Metrics
• Information divergences and likelihood ratios of Poisson processes and point patterns
• Machine Unlearning via Algorithmic Stability
• Quantifying dependencies for sensitivity analysis with multivariate input sample data
• Quantum Scalar Field Theory Based on the Extended Least Action Principle
• Supervising the Decoder of Variational Autoencoders to Improve Scientific Utility
• Some distance bounds of branching processes and their diffusion limits
• Relative Entropy via Distribution of Observables
• Taming numerical imprecision by adapting the KL divergence to negative probabilities
• Partition function approach to non-Gaussian likelihoods: Formalism and expansions for weakly non-Gaussian cosmological inference
• Molecular Noise In Synaptic Communication
• Asymptotics for Strassen's Optimal Transport Problem
• Divergence Regulated Encoder Network for Joint Dimensionality Reduction and Classification
• Variational Representations and Neural Network Estimation of Rényi Divergences
• Quantum natural gradient without monotonicity
• Universal Hypothesis Testing with Kernels: Asymptotically Optimal Tests for Goodness of Fit
• A unified view on differential privacy and robustness to adversarial examples
• On Exact and $\infty$-Rényi Common Informations
• Distributional Robustness and Uncertainty Quantification for Rare Events
• Generalized Entropic Quantum Speed Limits
• The Sphere Packing Bound for DSPCs with Feedback a la Augustin
• KALE Flow: A Relaxed KL Gradient Flow for Probabilities with Disjoint Support
• Hierarchical Micro-Macro Acceleration for Moment Models of Kinetic Equations
• Stability of the spectral gap and ground state indistinguishability for a decorated AKLT model
• Review, Analysis and Design of a Comprehensive Deep Reinforcement Learning Framework
• Towards Robust Off-Policy Evaluation via Human Inputs
• Rényi-Holevo inequality from $\alpha$-$z$-Rényi relative entropies
• Evolved Quantum Boltzmann Machines
• Better Heisenberg limits, coherence bounds, and energy-time tradeoffs via quantum Rényi information
• Relative Entropy and Mutual Information in Gaussian Statistical Field Theory
• One-shot information-theoretical approaches to fluctuation theorems
• All in the Exponential Family: Bregman Duality in Thermodynamic Variational Inference
• On nonlinear compression costs: when Shannon meets Rényi
• Precision-Recall Curves Using Information Divergence Frontiers
• Likelihood Ratio Exponential Families
• Mixture weights optimisation for Alpha-Divergence Variational Inference
• Private Stochastic Convex Optimization: Efficient Algorithms for Non-smooth Objectives
• Divergences induced by dual subtractive and divisive normalizations of exponential families and their convex deformations
• On The Chain Rule Optimal Transport Distance
• The Census and the Second Law: An Entropic Approach to Optimal Apportionment for the U.S. House of Representatives
• Phase space distributions in information theory
• Quantum Generative Training Using Rényi Divergences
• Faster Differentially Private Samplers via Rényi Divergence Analysis of Discretized Langevin MCMC
• MAC Resolvability: First And Second Order Results
• How Many Samples Required in Big Data Collection: A Differential Message Importance Measure
• The aBc Problem and Equator Sampling Renyi Divergences
• On conditional Sibson's $\alpha$-Mutual Information
• An information diffusion Fano inequality
• Semantic Security with Infinite Dimensional Quantum Eavesdropping Channel
• Variational Representations of Annealing Paths: Bregman Information under Monotonic Embedding
• Strong Asymptotic Composition Theorems for Mutual Information Measures
• Cramér-Rao Lower Bounds Arising from Generalized Csiszár Divergences
• Uncertainty-aware Sensitivity Analysis Using Rényi Divergences
• Asymptotically Optimal One- and Two-Sample Testing with Kernels
• Adaptive Quantum Generative Training using an Unbounded Loss Function
• Quantum speed limits based on Jensen-Shannon and Jeffreys divergences for general physical processes
• Asymptotically Optimal Tests for One- and Two-Sample Problems
• Transient fractality as a mechanism for emergent irreversibility in chaotic Hamiltonian dynamics
• Exponentially Consistent Kernel Two-Sample Tests
• Asymptotically Minimax Robust Hypothesis Testing
• Wyner's Common Information under Rényi Divergence Measures
• Learning Neural Models for Natural Language Processing in the Face of Distributional Shift
• Adaptive Importance Sampling meets Mirror Descent: a Bias-variance tradeoff
• Mind the Gap when Conditioning Amortised Inference in Sequential Latent-Variable Models
• Information Theoretic Inequalities as Bounds in Superconformal Field Theory
• A Beta-Beta Achievability Bound with Applications
• Optimal Posteriors for Chi-squared Divergence based PAC-Bayesian Bounds and Comparison with KL-divergence based Optimal Posteriors and Cross-Validation Procedure
• Continuous indetermination and average likelihood minimization
• Chaining Meets Chain Rule: Multilevel Entropic Regularization and Training of Neural Nets
• Alpha-Divergences in Variational Dropout
• Minimum KL-divergence on complements of $L_1$ balls
• Estimating Rényi's $\alpha$-Cross-Entropies in a Matrix-Based Way
• Multifractal of mass function
• Privacy-Utility Tradeoff for Hypothesis Testing Over A Noisy Channel
• Relations Between the Conditional Normalized Maximum Likelihood Distributions and the Latent Information Priors
• Simulation of Random Variables under Rényi Divergence Measures of All Orders
• A Novel Approach to Differential Privacy with Alpha Divergence
• Robust estimation of a Markov chain transition matrix from multiple sample paths
• An Information Bottleneck Problem with Rényi's Entropy
• BGC: Multi-Agent Group Belief with Graph Clustering
• PyLIT: Reformulation and implementation of the analytic continuation problem using kernel representation methods
• Optimal low-rank posterior mean and distribution approximation in linear Gaussian inverse problems on Hilbert spaces
• Towards Assessment of Randomized Smoothing Mechanisms for Certifying Adversarial Robustness
• Maximum Multiscale Entropy and Neural Network Regularization
• Alternative Framework to Quantize Fermionic Fields
• Privacy against a Hypothesis Testing Adversary
• Tunable Measures for Information Leakage and Applications to Privacy-Utility Tradeoffs
• Deep Samplable Observation Model for Global Localization and Kidnapping
• Intriguing Properties of Input-dependent Randomized Smoothing
• Tractable fish growth models considering individual differences with an application to the fish Plecoglossus altivelis altivelis
• Which Samples Should be Learned First: Easy or Hard?
• Comments on "Physical-layer cryptography through massive MIMO"
• Estimating Rationally Inattentive Utility Functions with Deep Clustering for Framing - Applications in YouTube Engagement Dynamics
• Riemannian-geometric generalizations of quantum fidelities and Bures-Wasserstein distance
• Information-theoretic applications of the logarithmic probability comparison bound
• Information Density Bounds for Privacy
• Capacity of Lorentzian polynomials and distance to binomial distributions
• On the convergence of the maximum likelihood estimator for the transition rate under a 2-state symmetric model
• PALMAR: Towards Adaptive Multi-inhabitant Activity Recognition in Point-Cloud Technology
• A Switch to the Concern of User: Importance Coefficient in Utility Distribution and Message Importance Measure
• Fluctuation Relations for Heat Exchange in the Generalized Gibbs Ensemble
• Boosting Certified Robustness for Time Series Classification with Efficient Self-Ensemble
• On Conditional $\alpha$-Information and its Application to Side-Channel Analysis
• Non-uniform Bounds in the Poisson Approximation with Applications to Informational Distances. I
• Differential Message Importance Measure: A New Approach to the Required Sampling Number in Big Data Structure Characterization
• End-to-End Simultaneous Dysarthric Speech Reconstruction with Frame-Level Adaptor and Multiple Wait-k Knowledge Distillation
• Probing information theoretic measures of nonlinear ultracold quantum gases using phase-space distributions
• Sharing quantum indistinguishability with multiple parties
• Markov chain entropy games and the geometry of their Nash equilibria
• Online Learning with Automata-based Expert Sequences
• Information geometry of nonmonotonic quantum natural gradient