Sharp Bounds Between Two Rényi Entropies of Distinct Positive Orders

Abstract

Many axiomatic definitions of entropy, such as the Rényi entropy, of a random variable are closely related to the ${\ell }_{\alpha }$-norm of its probability distribution. This study considers probability distributions on finite sets, and examines the sharp bounds of the ${\ell }_{\beta }$-norm with a fixed ${\ell }_{\alpha }$-norm, $\alpha \ne \beta$, for $n$-dimensional probability vectors with an integer $n\ge 2$. From the results, we derive the sharp bounds of the Rényi entropy of positive order $\beta$ with a fixed Rényi entropy of another positive order $\alpha$. As applications, we investigate sharp bounds of Ariomoto's mutual information of order $\alpha$ and Gallager's random coding exponents for uniformly focusing channels under the uniform input distribution.

A short version has been submitted to IEEE ITW2016. The short version is titled "Tight Bounds Between Two Rényi Entropies of Distinct Positive Orders."