Rényi Divergence and Kullback-Leibler Divergence

Abstract

Rényi divergence is related to Rényi entropy much like Kullback-Leibler divergence is related to Shannon's entropy, and comes up in many settings. It was introduced by Rényi as a measure of information that satisfies almost the same axioms as Kullback-Leibler divergence, and depends on a parameter that is called its order. In particular, the Rényi divergence of order 1 equals the Kullback-Leibler divergence. We review and extend the most important properties of Rényi divergence and Kullback-Leibler divergence, including convexity, continuity, limits of $\sigma$-algebras and the relation of the special order 0 to the Gaussian dichotomy and contiguity. We also show how to generalize the Pythagorean inequality to orders different from 1, and we extend the known equivalence between channel capacity and minimax redundancy to continuous channel inputs (for all orders) and present several other minimax results.

To appear in IEEE Transactions on Information Theory

References in corpus (2)

• From $\epsilon$-entropy to KL-entropy: Analysis of minimum information complexity density estimation
• Multiple Source Adaptation and the Renyi Divergence

Cited by in corpus (309)

• Renyi Differential Privacy
• The Perception-Distortion Tradeoff
• A review of domain adaptation without target labels
• Scalable Private Learning with PATE
• $f$-divergence Inequalities
• An introduction to domain adaptation and transfer learning
• Federated Learning with Bayesian Differential Privacy
• Differentially Private Model Publishing for Deep Learning
• On a generalization of the Jensen-Shannon divergence
• Fusion of Probability Density Functions
• Privacy Amplification by Iteration
• Dropout Inference in Bayesian Neural Networks with Alpha-divergences
• Information fractal dimension of mass function
• ordpy: A Python package for data analysis with permutation entropy and ordinal network methods
• Correlating thermal machines and the second law at the nanoscale
• Regimes of classical simulability for noisy Gaussian boson sampling
• Rényi Differential Privacy of the Sampled Gaussian Mechanism
• Estimating Mixture Entropy with Pairwise Distances
• On Variational Expressions for Quantum Relative Entropies
• Community detection in multi-relational data with restricted multi-layer stochastic blockmodel
• Stochastic independence as a resource in small-scale thermodynamics
• Beyond the entropy power inequality, via rearrangements
• Spectral MLE: Top-$K$ Rank Aggregation from Pairwise Comparisons
• Concentrated Differential Privacy: Simplifications, Extensions, and Lower Bounds
• Rethinking Lossy Compression: The Rate-Distortion-Perception Tradeoff
• Generalized Variational Inference: Three arguments for deriving new Posteriors
• Operational Interpretation of Renyi Information Measures via Composite Hypothesis Testing Against Product and Markov Distributions
• Guaranteed bounds on the Kullback-Leibler divergence of univariate mixtures using piecewise log-sum-exp inequalities
• Variational Inference of Disentangled Latent Concepts from Unlabeled Observations
• Optimal Quantization for Distribution Synthesis
• Consistency of Variational Bayes Inference for Estimation and Model Selection in Mixtures
• Generalization Bounds via Information Density and Conditional Information Density
• Generating Synthetic Mobility Networks with Generative Adversarial Networks
• On $f$-Divergences: Integral Representations, Local Behavior, and Inequalities
• Uniqueness and characterization theorems for generalized entropies
• A generalization of majorization that characterizes Shannon entropy
• Semantic Security via Seeded Modular Coding Schemes and Ramanujan Graphs
• Large-Scale Methods for Distributionally Robust Optimization
• Entropic Steering Criteria: Applications to Bipartite and Tripartite Systems
• OutbreakFlow: Model-based Bayesian inference of disease outbreak dynamics with invertible neural networks and its application to the COVID-19 pandemics in Germany
• Robust Hypothesis Testing with $\alpha$-Divergence
• Entropy and relative entropy from information-theoretic principles
• Characterization of Time Series Via Rényi Complexity-Entropy Curves
• On a Variational Definition for the Jensen-Shannon Symmetrization of Distances based on the Information Radius
• Formal Groups and $Z$-Entropies
• Information-theoretic bounds for exact recovery in weighted stochastic block models using the Renyi divergence
• The Renyi Capacity and Center
• Renyi relative entropies of quantum Gaussian states
• Lower Bounds for Locally Private Estimation via Communication Complexity
• Exact solution for the quantum and private capacities of bosonic dephasing channels
• Defining quantum divergences via convex optimization
• Information Aware Max-Norm Dirichlet Networks for Predictive Uncertainty Estimation
• GS-WGAN: A Gradient-Sanitized Approach for Learning Differentially Private Generators
• Bounding generalized relative entropies: Nonasymptotic quantum speed limits
• A Non-Asymptotic Analysis for Stein Variational Gradient Descent
• Domain Adaptation and Autoencoder Based Unsupervised Speech Enhancement
• Pointwise Maximal Leakage
• Relative Entropy and Catalytic Relative Majorization
• Scalable Information Inequalities for Uncertainty Quantification
• Approximate Span Liftings
• On the Kullback-Leibler divergence between discrete normal distributions
• Gibbs posterior concentration rates under sub-exponential type losses
• Decentralized and Model-Free Federated Learning: Consensus-Based Distillation in Function Space
• Differentially Private Federated Learning on Heterogeneous Data
• Evaluating Model Robustness and Stability to Dataset Shift
• Quantum work statistics and resource theories: bridging the gap through Renyi divergences
• The Augustin Capacity and Center
• Holographic second laws of black hole thermodynamics
• Coding for Positive Rate in the Source Model Key Agreement Problem
• On The Classification-Distortion-Perception Tradeoff
• Toward physical realizations of thermodynamic resource theories
• Subsampling Generative Adversarial Networks: Density Ratio Estimation in Feature Space with Softplus Loss
• Contagion in financial systems: A Bayesian network approach
• Relations between Dissipated Work and Rényi Divergences
• On Relations Between the Relative entropy and ${\chi }^2$-Divergence, Generalizations and Applications
• Alpha-Beta Divergence For Variational Inference
• alpha-Deep Probabilistic Inference (alpha-DPI): efficient uncertainty quantification from exoplanet astrometry to black hole feature extraction
• Improved bound on entropy production in a quantum annealer
• Revisiting Chernoff Information with Likelihood Ratio Exponential Families
• Properties of Noncommutative Renyi and Augustin Information
• Optimal Experimental Design Using A Consistent Bayesian Approach
• Analysis of Langevin Monte Carlo via convex optimization
• DPIS: An Enhanced Mechanism for Differentially Private SGD with Importance Sampling
• Internal entanglement and external correlations of any form limit each other
• Work fluctuation-dissipation trade-off in heat engines
• Thermodynamic utility of Non-Markovianity from the perspective of resource interconversion
• On Data-Processing and Majorization Inequalities for $f$-Divergences with Applications
• Infomax strategies for an optimal balance between exploration and exploitation
• Exponential decay of Rényi divergence under Fokker-Planck equations
• Reasoning About Generalization via Conditional Mutual Information
• Maxwell's Demon walks into Wall Street: Stochastic Thermodynamics meets Expected Utility Theory
• Quantifying non-Gaussianity of a quantum state by the negative entropy of quadrature distributions
• On-Average KL-Privacy and its equivalence to Generalization for Max-Entropy Mechanisms
• Riesz-Quincunx-UNet Variational Auto-Encoder for Satellite Image Denoising
• The variance of relative surprisal as single-shot quantifier
• InfoAT: Improving Adversarial Training Using the Information Bottleneck Principle
• Data-driven priors and their posterior concentration rates
• A Joint Typicality Approach to Algebraic Network Information Theory
• Pretty good quantum state transfer on isotropic and anisotropic Heisenberg spin chains with tailored site dependent exchange couplings
• Autofocused oracles for model-based design
• The Sphere Packing Bound For Memoryless Channels
• Two Measures of Dependence
• Autonomous Quantum Devices: When Are They Realizable without Additional Thermodynamic Costs?
• A Scale-invariant Generalization of the Rényi Entropy, Associated Divergences and their Optimizations under Tsallis' Nonextensive Framework
• Least $k$th-Order and Rényi Generative Adversarial Networks
• Consistency of ELBO maximization for model selection
• Amplifying Inter-message Distance: On Information Divergence Measures in Big Data
• Characterisation of quantum betting tasks in terms of Arimoto mutual information
• Information-Theoretic Perspectives on Brascamp-Lieb Inequality and Its Reverse
• Contextual Distillation Model for Diversified Recommendation
• Proximal Langevin Algorithm: Rapid Convergence Under Isoperimetry
• Importance of Small Probability Events in Big Data: Information Measures, Applications, and Challenges
• Generalization Error Bounds Via Rényi-, $f$-Divergences and Maximal Leakage
• Refined Young inequality and its application to divergences
• Optimal Accounting of Differential Privacy via Characteristic Function
• The Sphere Packing Bound via Augustin's Method
• Asymmetry and tighter uncertainty relations for Rényi entropies via quantum-classical decompositions of resource measures
• Mosaics of combinatorial designs for information-theoretic security
• Transport-majorization to analytic and geometric inequalities
• On Reverse Pinsker Inequalities
• Tight Bounds for Symmetric Divergence Measures and a Refined Bound for Lossless Source Coding
• Scalable Variational Inference in Log-supermodular Models
• Quantum Foundations of Classical Reversible Computing
• The Reconfiguration Pattern of Individual Brain Metabolic Connectome for Parkinson's Disease Identification
• Concurrent Composition Theorems for Differential Privacy
• Refined Strong Converse for the Constant Composition Codes
• Methods for Quantifying Dataset Similarity: a Review, Taxonomy and Comparison
• Infinite-dimensional gradient-based descent for alpha-divergence minimisation
• Variational f-divergence Minimization
• Convergence Rates of Variational Posterior Distributions
• Markov Categories and Entropy
• Pufferfish Privacy: An Information-Theoretic Study
• DOCTOR: A Simple Method for Detecting Misclassification Errors
• Duality between source coding with quantum side information and c-q channel coding
• Multiple Private Key Generation for Continuous Memoryless Sources with A Helper
• The MAC Resolvability Region, Semantic Security and Its Operational Implications
• Bayesian Differential Privacy for Machine Learning
• Leave Zero Out: Towards a No-Cross-Validation Approach for Model Selection
• Links between Dissipation and Rényi Divergences in $\mathcal{P}\mathcal{T}$-Symmetric Quantum Mechanics
• DistFL: Distribution-aware Federated Learning for Mobile Scenarios
• An empirical $G$-Wishart prior for sparse high-dimensional Gaussian graphical models
• Maximum one-shot dissipated work from Renyi divergences
• Strong Converse Theorems for Classes of Multimessage Multicast Networks: A Rényi Divergence Approach
• Generalized Mixability via Entropic Duality
• Effective Sample Size, Dimensionality, and Generalization in Covariate Shift Adaptation
• Relating relative Rényi entropies and Wigner-Yanase-Dyson skew information to generalized multiple quantum coherences
• A Simple Derivation of the Refined Sphere Packing Bound Under Certain Symmetry Hypotheses
• A Mathematical Theory of Attention
• Relations between Heat Exchange and Rényi Divergences
• The Effect of Quantization in Federated Learning: A Rényi Differential Privacy Perspective
• On the Support Recovery of Jointly Sparse Gaussian Sources using Sparse Bayesian Learning
• Speed limits and thermodynamic uncertainty relations for quantum systems governed by non-Hermitian Hamiltonian
• Point Information Gain and Multidimensional Data Analysis
• Quantum Channel Learning
• Bayesian fractional posteriors
• Multi-Cause Deconfounding for Recommender Systems with Latent Confounders
• Minimax Rényi Redundancy
• A Bounded Measure for Estimating the Benefit of Visualization: Theoretical Discourse and Conceptual Evaluation
• Operator-valued Schatten spaces and quantum entropies
• Estimation of KL Divergence: Optimal Minimax Rate
• Conditional Rényi entropy and the relationships between Rényi capacities
• Differentially Private Confidence Intervals for Proportions under Stratified Random Sampling
• A Bounded Measure for Estimating the Benefit of Visualization
• Renyi divergences from Euclidean quenches
• Rationally Inattentive Inverse Reinforcement Learning Explains YouTube Commenting Behavior
• Stochastic Proximal Langevin Algorithm: Potential Splitting and Nonasymptotic Rates
• Catalytic transformations for thermal operations
• A New Approach to Adaptive Data Analysis and Learning via Maximal Leakage
• A precise bare simulation approach to the minimization of some distances. Foundations
• Rényi divergence and the central limit theorem
• Hyperparameter Tuning with Renyi Differential Privacy
• Strongly Convex Divergences
• Conditions for the existence of a generalization of Rényi divergence
• Resolvability on Continuous Alphabets
• Variational approximations using Fisher divergence
• Complex-valued Federated Learning with Differential Privacy and MRI Applications
• On the Existence of the Augustin Mean
• Dissipation in the Generalized Gibbs Ensemble
• Properties of a Generalized Divergence Related to Tsallis Relative Entropy
• Concentration properties of fractional posterior in 1-bit matrix completion
• Matching Users' Preference Under Target Revenue Constraints in Optimal Data Recommendation Systems
• The Augustin Center and The Sphere Packing Bound For Memoryless Channels
• Strong Brascamp-Lieb Inequalities
• Bounding the resources for thermalizing many-body localized systems
• Generalizations of Fano's Inequality for Conditional Information Measures via Majorization Theory
• Equivocations, Exponents and Second-Order Coding Rates under Various Rényi Information Measures
• Locally-Measured Rényi Divergences
• Towards the Unification and Robustness of Perturbation and Gradient Based Explanations
• AED-PADA:Improving Generalizability of Adversarial Example Detection via Principal Adversarial Domain Adaptation
• High Resolution Face Editing with Masked GAN Latent Code Optimization
• Quantum $f$-divergences via Nussbaum-Szkoła Distributions and Applications to $f$-divergence Inequalities
• Always Lurking: Understanding and Mitigating Bias in Online Human Trafficking Detection
• Calibrating Noise to Variance in Adaptive Data Analysis
• Concentration of a sparse Bayesian model with Horseshoe prior in estimating high-dimensional precision matrix
• Semi-Supervised Hypothesis Transfer for Source-Free Domain Adaptation
• Information divergences and likelihood ratios of Poisson processes and point patterns
• Machine Unlearning via Algorithmic Stability
• Relative Entropy via Distribution of Observables
• Random Matrix Improved Covariance Estimation for a Large Class of Metrics
• Quantifying dependencies for sensitivity analysis with multivariate input sample data
• Speed limit, dissipation bound and dissipation-time trade-off in thermal relaxation processes
• Quantum Scalar Field Theory Based on the Extended Least Action Principle
• Taming numerical imprecision by adapting the KL divergence to negative probabilities
• Point Divergence Gain and Multidimensional Data Sequences Analysis
• Storage Space Allocation Strategy for Digital Data with Message Importance
• Contextual Fine-to-Coarse Distillation for Coarse-grained Response Selection in Open-Domain Conversations
• Supervising the Decoder of Variational Autoencoders to Improve Scientific Utility
• Generalized Entropic Quantum Speed Limits
• Stability of the spectral gap and ground state indistinguishability for a decorated AKLT model
• Quantum natural gradient without monotonicity
• A unified view on differential privacy and robustness to adversarial examples
• Universal Hypothesis Testing with Kernels: Asymptotically Optimal Tests for Goodness of Fit
• KALE Flow: A Relaxed KL Gradient Flow for Probabilities with Disjoint Support
• Distributional Robustness and Uncertainty Quantification for Rare Events
• Review, Analysis and Design of a Comprehensive Deep Reinforcement Learning Framework
• Hierarchical Micro-Macro Acceleration for Moment Models of Kinetic Equations
• On Exact and $\infty$-Rényi Common Informations
• Divergence Regulated Encoder Network for Joint Dimensionality Reduction and Classification
• Variational Representations and Neural Network Estimation of Rényi Divergences
• Partition function approach to non-Gaussian likelihoods: Formalism and expansions for weakly non-Gaussian cosmological inference
• The Sphere Packing Bound for DSPCs with Feedback a la Augustin
• Asymptotics for Strassen's Optimal Transport Problem
• Molecular Noise In Synaptic Communication
• Relative Entropy and Mutual Information in Gaussian Statistical Field Theory
• Evolved Quantum Boltzmann Machines
• All in the Exponential Family: Bregman Duality in Thermodynamic Variational Inference
• MAC Resolvability: First And Second Order Results
• How Many Samples Required in Big Data Collection: A Differential Message Importance Measure
• Better Heisenberg limits, coherence bounds, and energy-time tradeoffs via quantum Rényi information
• One-shot information-theoretical approaches to fluctuation theorems
• Quantum Generative Training Using Rényi Divergences
• The Census and the Second Law: An Entropic Approach to Optimal Apportionment for the U.S. House of Representatives
• Faster Differentially Private Samplers via Rényi Divergence Analysis of Discretized Langevin MCMC
• Private Stochastic Convex Optimization: Efficient Algorithms for Non-smooth Objectives
• Towards Robust Off-Policy Evaluation via Human Inputs
• Mixture weights optimisation for Alpha-Divergence Variational Inference
• Divergences induced by dual subtractive and divisive normalizations of exponential families and their convex deformations
• On The Chain Rule Optimal Transport Distance
• Likelihood Ratio Exponential Families
• Rényi-Holevo inequality from $\alpha$-$z$-Rényi relative entropies
• Phase space distributions in information theory
• On nonlinear compression costs: when Shannon meets Rényi
• Precision-Recall Curves Using Information Divergence Frontiers
• Quantum speed limits based on Jensen-Shannon and Jeffreys divergences for general physical processes
• Adaptive Quantum Generative Training using an Unbounded Loss Function
• Variational Representations of Annealing Paths: Bregman Information under Monotonic Embedding
• Semantic Security with Infinite Dimensional Quantum Eavesdropping Channel
• Asymptotically Optimal Tests for One- and Two-Sample Problems
• Continuous indetermination and average likelihood minimization
• On conditional Sibson's $\alpha$-Mutual Information
• Learning Neural Models for Natural Language Processing in the Face of Distributional Shift
• Strong Asymptotic Composition Theorems for Mutual Information Measures
• Cramér-Rao Lower Bounds Arising from Generalized Csiszár Divergences
• Adaptive Importance Sampling meets Mirror Descent: a Bias-variance tradeoff
• Uncertainty-aware Sensitivity Analysis Using Rényi Divergences
• Mind the Gap when Conditioning Amortised Inference in Sequential Latent-Variable Models
• Asymptotically Optimal One- and Two-Sample Testing with Kernels
• An information diffusion Fano inequality
• Optimal Posteriors for Chi-squared Divergence based PAC-Bayesian Bounds and Comparison with KL-divergence based Optimal Posteriors and Cross-Validation Procedure
• Transient fractality as a mechanism for emergent irreversibility in chaotic Hamiltonian dynamics
• Exponentially Consistent Kernel Two-Sample Tests
• Asymptotically Minimax Robust Hypothesis Testing
• Wyner's Common Information under Rényi Divergence Measures
• Minimum KL-divergence on complements of $L_1$ balls
• The aBc Problem and Equator Sampling Renyi Divergences
• Information Theoretic Inequalities as Bounds in Superconformal Field Theory
• Chaining Meets Chain Rule: Multilevel Entropic Regularization and Training of Neural Nets
• A Beta-Beta Achievability Bound with Applications
• Alpha-Divergences in Variational Dropout
• Differential Message Importance Measure: A New Approach to the Required Sampling Number in Big Data Structure Characterization
• Simulation of Random Variables under Rényi Divergence Measures of All Orders
• Privacy against a Hypothesis Testing Adversary
• Tunable Measures for Information Leakage and Applications to Privacy-Utility Tradeoffs
• Maximum Multiscale Entropy and Neural Network Regularization
• Towards Assessment of Randomized Smoothing Mechanisms for Certifying Adversarial Robustness
• On the convergence of the maximum likelihood estimator for the transition rate under a 2-state symmetric model
• An Information Bottleneck Problem with Rényi's Entropy
• Non-uniform Bounds in the Poisson Approximation with Applications to Informational Distances. I
• Robust estimation of a Markov chain transition matrix from multiple sample paths
• Multifractal of mass function
• Estimating Rényi's $\alpha$-Cross-Entropies in a Matrix-Based Way
• Sharing quantum indistinguishability with multiple parties
• Privacy-Utility Tradeoff for Hypothesis Testing Over A Noisy Channel
• BGC: Multi-Agent Group Belief with Graph Clustering
• Deep Samplable Observation Model for Global Localization and Kidnapping
• Capacity of Lorentzian polynomials and distance to binomial distributions
• Information geometry of nonmonotonic quantum natural gradient
• On Conditional $\alpha$-Information and its Application to Side-Channel Analysis
• A Switch to the Concern of User: Importance Coefficient in Utility Distribution and Message Importance Measure
• A Novel Approach to Differential Privacy with Alpha Divergence
• PALMAR: Towards Adaptive Multi-inhabitant Activity Recognition in Point-Cloud Technology
• Which Samples Should be Learned First: Easy or Hard?
• Intriguing Properties of Input-dependent Randomized Smoothing
• Boosting Certified Robustness for Time Series Classification with Efficient Self-Ensemble
• End-to-End Simultaneous Dysarthric Speech Reconstruction with Frame-Level Adaptor and Multiple Wait-k Knowledge Distillation
• Probing information theoretic measures of nonlinear ultracold quantum gases using phase-space distributions
• Markov chain entropy games and the geometry of their Nash equilibria
• Information Density Bounds for Privacy
• Riemannian-geometric generalizations of quantum fidelities and Bures-Wasserstein distance
• Tractable fish growth models considering individual differences with an application to the fish Plecoglossus altivelis altivelis
• Alternative Framework to Quantize Fermionic Fields
• Optimal low-rank posterior mean and distribution approximation in linear Gaussian inverse problems on Hilbert spaces
• Relations Between the Conditional Normalized Maximum Likelihood Distributions and the Latent Information Priors
• Information-theoretic applications of the logarithmic probability comparison bound
• PyLIT: Reformulation and implementation of the analytic continuation problem using kernel representation methods
• Comments on "Physical-layer cryptography through massive MIMO"
• Estimating Rationally Inattentive Utility Functions with Deep Clustering for Framing - Applications in YouTube Engagement Dynamics
• Fluctuation Relations for Heat Exchange in the Generalized Gibbs Ensemble
• Online Learning with Automata-based Expert Sequences