First order online optimisation using forward gradients in over-parameterised systems

Behnam Mafakheri, Iman Shames, Jonathan H. Manton

The success of deep learning over the past decade mainly relies on gradient-based optimisation and backpropagation. This paper focuses on analysing the performance of first-order g…