Thermodynamic efficiency of learning a rule in neural networks

Abstract

Biological systems have to build models from their sensory data that allow them to efficiently process previously unseen inputs. Here, we study a neural network learning a linearly separable rule using examples provided by a teacher. We analyse the ability of the network to apply the rule to new inputs, that is to generalise from past experience. Using stochastic thermodynamics, we show that the thermodynamic costs of the learning process provide an upper bound on the amount of information that the network is able to learn from its teacher for both batch and online learning. This allows us to introduce a thermodynamic efficiency of learning. We analytically compute the dynamics and the efficiency of a noisy neural network performing online learning in the thermodynamic limit. In particular, we analyse three popular learning algorithms, namely Hebbian, Perceptron and AdaTron learning. Our work extends the methods of stochastic thermodynamics to a new type of learning problem and might form a suitable basis for investigating the thermodynamics of decision-making.

arXiv comment10 pages, 6 figures; 7 pages of supplemental material with one figure

References in corpus (19)

• Stochastic thermodynamics, fluctuation theorems, and molecular machines
• Steady State Thermodynamics of Langevin Systems
• Statistical physics of inference: Thresholds and algorithms
• The Energetic Costs of Cellular Computation
• Flow Navigation by Smart Microswimmers via Reinforcement Learning
• Thermodynamics with continuous information flow
• The Three Faces of the Second Law: II. Fokker-Planck Formulation
• Stochastic thermodynamics of bipartite systems: transfer entropy inequalities and a Maxwell's demon interpretation
• Thermodynamic costs of information processing in sensory adaption
• Efficiency of cellular information processing
• Maxwell's demon in biochemical signal transduction with feedback loop
• Maximum likelihood and the single receptor
• Thermodynamics of statistical inference by cells
• Thermodynamic efficiency of information and heat flow
• Information-theoretic vs. thermodynamic entropy production in autonomous sensory networks
• Multipartite information flow for multiple Maxwell demons
• Chemical sensing by nonequilibrium cooperative receptors
• Stochastic Thermodynamics of Learning
• Nonequilibrium Thermodynamics of Restricted Boltzmann Machines

Cited by in corpus (4)

• Effective learning is accompanied by high dimensional and efficient representations of neural activity
• A Physical Quantum Agent
• The Physics of Learning
• Nonequilibrium thermodynamics of self-supervised learning