Stochastic Thermodynamics of Learning Parametric Probabilistic Models

Abstract

We have formulated a family of machine learning problems as the time evolution of Parametric Probabilistic Models (PPMs), inherently rendering a thermodynamic process. Our primary motivation is to leverage the rich toolbox of thermodynamics of information to assess the information-theoretic content of learning a probabilistic model. We first introduce two information-theoretic metrics: Memorized-information (M-info) and Learned-information (L-info), which trace the flow of information during the learning process of PPMs. Then, we demonstrate that the accumulation of L-info during the learning process is associated with entropy production, and parameters serve as a heat reservoir in this process, capturing learned information in the form of M-info.

References in corpus (12)

• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• The entropy of Hawking radiation
• Entropy production as correlation between system and reservoir
• Ensemble and Trajectory Thermodynamics: A Brief Introduction
• Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Data
• The thermodynamics of prediction
• Fluctuation Theorem with Information Exchange: Role of Correlations in Stochastic Thermodynamics
• Langevin equation with colored noise for constant-temperature molecular dynamics simulations
• Thermodynamic and Logical Reversibilities Revisited
• Local detailed balance
• Tightening Mutual Information Based Bounds on Generalization Error
• Detailed Fluctuation Theorems: A Unifying Perspective