Thermodynamic efficiency of learning a rule in neural networks
arXiv:1706.09713 · doi:10.1088/1367-2630/aa89ff
Abstract
Biological systems have to build models from their sensory data that allow them to efficiently process previously unseen inputs. Here, we study a neural network learning a linearly separable rule using examples provided by a teacher. We analyse the ability of the network to apply the rule to new inputs, that is to generalise from past experience. Using stochastic thermodynamics, we show that the thermodynamic costs of the learning process provide an upper bound on the amount of information that the network is able to learn from its teacher for both batch and online learning. This allows us to introduce a thermodynamic efficiency of learning. We analytically compute the dynamics and the efficiency of a noisy neural network performing online learning in the thermodynamic limit. In particular, we analyse three popular learning algorithms, namely Hebbian, Perceptron and AdaTron learning. Our work extends the methods of stochastic thermodynamics to a new type of learning problem and might form a suitable basis for investigating the thermodynamics of decision-making.
10 pages, 6 figures; 7 pages of supplemental material with one figure
References in corpus (7)
- Thermodynamic costs of information processing in sensory adaption
- Efficiency of cellular information processing
- Maxwell's demon in biochemical signal transduction with feedback loop
- Thermodynamics of statistical inference by cells
- Information-theoretic vs. thermodynamic entropy production in autonomous sensory networks
- Multipartite information flow for multiple Maxwell demons
- Stochastic Thermodynamics of Learning