Estimating Rényi's $\alpha$-Cross-Entropies in a Matrix-Based Way

Abstract

Conventional information-theoretic quantities assume access to probability distributions. Estimating such distributions is not trivial. Here, we consider function-based formulations of cross entropy that sidesteps this a priori estimation requirement. We propose three measures of Rényi's $\alpha$-cross-entropies in the setting of reproducing-kernel Hilbert spaces. Each measure has its appeals. We prove that we can estimate these measures in an unbiased, non-parametric, and minimax-optimal way. We do this via sample-constructed Gram matrices. This yields matrix-based estimators of Rényi's $\alpha$-cross-entropies. These estimators satisfy all of the axioms that Rényi established for divergences. Our cross-entropies can thus be used for assessing distributional differences. They are also appropriate for handling high-dimensional distributions, since the convergence rate of our estimator is independent of the sample dimensionality. Python code for implementing these measures can be found at https://github.com/isledge/MBRCE

arXiv commentSubmitted to the IEEE Transactions on Information Theory

References in corpus (17)

• Rényi Divergence and Kullback-Leibler Divergence
• Equivalence of distance-based and RKHS-based statistics in hypothesis testing
• Quantum conditional mutual information and approximate Markov chains
• Sandwiched Rényi Divergence Satisfies Data Processing Inequality
• Quantum hypothesis testing and the operational interpretation of the quantum Renyi relative entropies
• Quantum f-divergences and error correction
• On the quantum Renyi relative entropies and related capacity formulas
• Estimation of Rényi Entropy and Mutual Information Based on Generalized Nearest-Neighbor Graphs
• Strengthened Monotonicity of Relative Entropy via Pinched Petz Recovery Map
• Characteristic and Universal Tensor Product Kernels
• A Kernel Independence Test for Random Processes
• Optimally-Weighted Herding is Bayesian Quadrature
• A Kernel Test for Three-Variable Interactions
• A Wild Bootstrap for Degenerate Kernel Tests
• Convergence guarantees for kernel-based quadrature rules in misspecified settings
• A lower bound of quantum conditional mutual information
• How not to Rényi generalize the Quantum Conditional Mutual Information