Linear KL-Optimal Frequency Normalisation

Abstract

Fast implementations of range coding and asymmetric numeral systems (ANS) owe their excellent performance to replacing slow division instructions by bit-shifts in their encoding and decoding algorithms. This is possible when the frequency distribution of symbols is normalised such that it sums to a power of two. However, such normalisation typically introduces a marginal increase in the Kullback-Leibler divergence between the original and the normalised distribution, leading to a worse compression ratio. We show that the currently used methods for frequency normalisation are suboptimal in both their running time and the achieved Kullback-Leibler divergence. We propose a new method for frequency normalisation that is asymptotically linear in the number of symbols and achieves the smallest possible Kullback-Leibler divergence between the original and the normalised distribution. The method is based on a solution to a separable concave optimisation problem, which may be of independent interest.

4 pages