Gibbs randomness-compression proposition
arXiv:2505.23869
The paper proposes a theorem linking Gibbs entropy (a measure of randomness) to lossy model compression, showing that the entropy of remaining network weights correlates with learning performance across several pruning and compressed‑sensing based compression methods.
Abstract
A proposition that connects randomness and compression is put forward via Gibbs entropy over set of measurement vectors associated with a lossy compression process. In building this connection, we use a performance of a learning task as a probe of compression in iterative compress-train cycles. This can be thought as iterative coarse-graining from statistical mechanics perspective using thermodynamic efficiency as a probe. We formulate this connection via comonotonic relationship within a very small decrease in compression ratio and the performance. We have showcase the validity of this proposition with a canonical vision task in deep learning with three different model compression processes as {\it a baseline model}. We use the following, simpler to more complex model compression approaches: (1) random pruning,(2) magnitude pruning, and (3) a more complex compression by using dual tomographic compression, which utilizes compressed sensing in dual fashion which is introduced as a new method. We use remaining weights of deep learning network as a measurement vector where we measure the Gibbs entropy. We show case the idea that there is an inherent computable connection between compression probed by performance and randomness from an entropy measure on the learned model.
10 pages, 5 figures, 2 tables, 1 algorithm pseudocode. New references, text improvements and concept mapping for terms. Codes are available on Zenodo repository: https://zenodo.org/records/15751974