Functional Optimisation of Online Algorithms in Multilayer Neural Networks

Abstract

We study the online dynamics of learning in fully connected soft committee machines in the student-teacher scenario. The locally optimal modulation function, which determines the learning algorithm, is obtained from a variational argument in such a manner as to maximise the average generalisation error decay per example. Simulations results for the resulting algorithm are presented for a few cases. The symmetric phase plateaux are found to be vastly reduced in comparison to those found when online backpropagation algorithms are used. A discussion of the implementation of these ideas as practical algorithms is given.

Cited by in corpus (6)

• Hidden Unit Specialization in Layered Neural Networks: ReLU vs. Sigmoidal Activation
• Analysis of Natural Gradient Descent for Multilayer Neural Networks
• Phase transitions in soft-committee machines
• Dynamical transitions in the evolution of learning algorithms by selection
• Supervised Learning in the Presence of Concept Drift: A modelling framework
• On-line learning dynamics of ReLU neural networks using statistical physics techniques