Weighted information and entropy rates

Abstract

The weighted entropy $H_{\varphi }^{\mathrm{w}}(X)=H_{\varphi }^{\mathrm{w}}(f)$ of a random variable $X$ with values $x$ and a probability-mass/density function $f$ is defined as the mean value $\mathbb{E}{I}_{\varphi }^{\mathrm{w}}(X)$ of the weighted information $I_{\varphi }^{\mathrm{w}}(x)=-\varphi (x)\log f(x)$. Here $x\mapsto \varphi (x)\in \mathbb{R}$ is a given weight function (WF) indicating a 'value' of outcome $x$. For an $n$-component random vector ${\mathbf{X}}_0^{n-1}=(X_0,\dots ,X_{n-1})$ produced by a random process $\mathbf{X}=(X_i,i\in \mathbb{Z})$, the weighted information $I_{{\varphi }_n}^{\mathrm{w}}({\mathbf{x}}_0^{n-1})$ and weighted entropy $H_{{\varphi }_n}^{\mathrm{w}}({\mathbf{X}}_0^{n-1})$ are defined similarly, with an WF ${\varphi }_n({\mathbf{x}}_0^{n-1})$. Two types of WFs ${\varphi }_n$ are considered, based on additive and a multiplicative forms (${\varphi }_n({\mathbf{x}}_0^{n-1})=\sum _{i=0}^{n-1}\phi (x_i)$ and ${\varphi }_n({\mathbf{x}}_0^{n-1})=\prod _{i=0}^{n-1}\phi (x_i)$, respectively). The focus is upon $rates$ of the weighted entropy and information, regarded as parameters related to $\mathbf{X}$. We show that, in the context of ergodicity, a natural scale for an asymptotically additive/multiplicative WF is $\frac{1}{n^2}{H}_{{\varphi }_n}^{\mathrm{w}}({\mathbf{X}}_0^{n-1})$ and $\frac{1}{n}\log H_{{\varphi }_n}^{\mathrm{w}}({\mathbf{X}}_0^{n-1})$, respectively. This gives rise to $primary$ $rates$. The next-order terms can also be identified, leading to $secondary$ $rates$. We also consider emerging generalisations of the Shannon-McMillan-Breiman theorem.

References in corpus (1)

• Weighted Gaussian entropy and determinant inequalities

Cited by in corpus (1)

• Weighted entropy and optimal portfolios for risk-averse Kelly investments