Critical Slowing Down Near Topological Transitions in Rate-Distortion Problems
arXiv:2103.02646 · doi:10.1109/ISIT45174.2021.9517956
Abstract
In rate-distortion (RD) problems one seeks reduced representations of a source that meet a target distortion constraint. Such optimal representations undergo topological transitions at some critical rate values, when their cardinality or dimensionality change. We study the convergence time of the Arimoto-Blahut alternating projection algorithms, used to solve such problems, near those critical points, both for the rate-distortion and information bottleneck settings. We argue that they suffer from critical slowing down -- a diverging number of iterations for convergence -- near the critical points. This phenomenon can have theoretical and practical implications for both machine learning and data compression problems.
arXiv comment10 pages, 2 figures, ISIT 2021 submission
References in corpus (6)
- The information bottleneck method
- Opening the Black Box of Deep Neural Networks via Information
- Squeezing the Arimoto-Blahut algorithm for faster convergence
- Relevance in the Renormalization Group and in Information Theory
- Phase Transitions for the Information Bottleneck in Representation Learning
- Analysis of the Convergence Speed of the Arimoto-Blahut Algorithm by the Second Order Recurrence Formula