The Entropy of Floating-Point Numbers

Abstract

Here we present an analytic approximation for the entropy of floating-point numbers, along with bounds on the error of this approximation. It is well-known that the differential entropy is tightly linked to the discrete entropy of a uniformly quantized random variable. Our approximation uncovers a different quantity that provides this link for floating-point quantization. Additionally, we prove that the entropy of a floating-point quantized random variable is approximately unchanged under scaling. Closed-form expressions for the floating-point entropy of common distributions are provided and compared to exact results.

17 pages, 12 figures. Full version of the paper accepted at ITW 2026. This update adds additional references, improves the introduction, tightens the lower bound on the KL divergence, and fixes typos