Stochastic Thermodynamics of Learning

Abstract

Virtually every organism gathers information about its noisy environment and builds models from that data, mostly using neural networks. Here, we use stochastic thermodynamics to analyse the learning of a classification rule by a neural network. We show that the information acquired by the network is bounded by the thermodynamic cost of learning and introduce a learning efficiency $\eta \le 1$. We discuss the conditions for optimal learning and analyse Hebbian learning in the thermodynamic limit.

5 pages, 3 figures, 7 pages of supplemental material

References in corpus (9)

• Thermodynamic uncertainty relation for biomolecular processes
• Extracting work from a single heat bath through feedback
• Thermodynamic costs of information processing in sensory adaption
• Efficiency of cellular information processing
• Maxwell's demon in biochemical signal transduction with feedback loop
• Second-law-like inequalities with information and their interpretations
• Thermodynamics of statistical inference by cells
• Multipartite information flow for multiple Maxwell demons
• Nonequilibrium sensing and its analogy to kinetic proofreading

Cited by in corpus (7)

• Deep Learning Theory Review: An Optimal Control and Dynamical Systems Perspective
• Uncertainty relations for time-delayed Langevin systems
• Quantifying Relevance in Learning and Inference
• Effective thermodynamics of two interacting underdamped Brownian particles
• Thermodynamic efficiency of learning a rule in neural networks
• Dissipation in non-steady state regulatory circuits
• Nonequilibrium thermodynamics of self-supervised learning