Statistical mechanics of continual learning: variational principle and mean-field potential

Abstract

An obstacle to artificial general intelligence is set by continual learning of multiple tasks of different nature. Recently, various heuristic tricks, both from machine learning and from neuroscience angles, were proposed, but they lack a unified theory ground. Here, we focus on continual learning in single-layered and multi-layered neural networks of binary weights. A variational Bayesian learning setting is thus proposed, where the neural networks are trained in a field-space, rather than gradient-ill-defined discrete-weight space, and furthermore, weight uncertainty is naturally incorporated, and modulates synaptic resources among tasks. From a physics perspective, we translate the variational continual learning into Franz-Parisi thermodynamic potential framework, where previous task knowledge acts as a prior and a reference as well. We thus interpret the continual learning of the binary perceptron in a teacher-student setting as a Franz-Parisi potential computation. The learning performance can then be analytically studied with mean-field order parameters, whose predictions coincide with numerical experiments using stochastic gradient descent methods. Based on the variational principle and Gaussian field approximation of internal preactivations in hidden layers, we also derive the learning algorithm considering weight uncertainty, which solves the continual learning with binary weights using multi-layered neural networks, and performs better than the currently available metaplasticity algorithm. Our proposed principled frameworks also connect to elastic weight consolidation, weight-uncertainty modulated learning, and neuroscience inspired metaplasticity, providing a theory-grounded method for the real-world multi-task learning with deep networks.

arXiv comment48 pages, 8 figures, final version to Phys Rev E

References in corpus (9)

• Overcoming catastrophic forgetting in neural networks
• Recipes for metastable states in Spin Glasses
• Alleviating catastrophic forgetting using context-dependent gating and synaptic stabilization
• Synaptic metaplasticity in binarized neural networks
• Origin of the computational hardness for learning with binary synapses
• On the role of synaptic stochasticity in training low-precision neural networks
• Learning credit assignment
• Phase Transitions in Transfer Learning for High-Dimensional Perceptrons
• Variational mean-field theory for training restricted Boltzmann machines with binary synapses

Cited by in corpus (2)

• Eight challenges in developing theory of intelligence
• Dynamics of Meta-learning Representation in the Teacher-student Scenario