On entropy for mixtures of discrete and continuous variables

Abstract

Let $X$ be a discrete random variable with support $S$ and $f:S\to S^{\prime }$ be a bijection. Then it is well-known that the entropy of $X$ is the same as the entropy of $f(X)$. This entropy preservation property has been well-utilized to establish non-trivial properties of discrete stochastic processes, e.g. queuing process \cite{prg03}. Entropy as well as entropy preservation is well-defined only in the context of purely discrete or continuous random variables. However for a mixture of discrete and continuous random variables, which arise in many interesting situations, the notions of entropy and entropy preservation have not been well understood. In this paper, we extend the notion of entropy in a natural manner for a mixed-pair random variable, a pair of random variables with one discrete and the other continuous. Our extensions are consistent with the existing definitions of entropy in the sense that there exist natural injections from discrete or continuous random variables into mixed-pair random variables such that their entropy remains the same. This extension of entropy allows us to obtain sufficient conditions for entropy preservation in mixtures of discrete and continuous random variables under bijections. The extended definition of entropy leads to an entropy rate for continuous time Markov chains. As an application, we recover a known probabilistic result related to Poisson process. We strongly believe that the frame-work developed in this paper can be useful in establishing probabilistic properties of complex processes, such as load balancing systems, queuing network, caching algorithms.

10 pages, two-column

Cited by in corpus (9)

• Integrating Artificial Intelligence with Real-time Intracranial EEG Monitoring to Automate Interictal Identification of Seizure Onset Zones in Focal Epilepsy
• Existence and Continuity of Differential Entropy for a Class of Distributions
• A probabilistic view of wave-particle duality for single photons
• Sequential- and Parallel- Constrained Max-value Entropy Search via Information Lower Bound
• Bounds on mutual information of mixture data for classification tasks
• Information-Theoretic Approach to Navigation for Efficient Detection and Classification of Underwater Objects
• Model of Cognitive Dynamics Predicts Performance on Standardized Tests
• On mutual information estimation for mixed-pair random variables
• Towards Verified Stochastic Variational Inference for Probabilistic Programs