On the biological plausibility of orthogonal initialisation for solving gradient instability in deep neural networks
arXiv:2211.08408 · doi:10.1109/ISCMI56532.2022.10068489
Abstract
Initialising the synaptic weights of artificial neural networks (ANNs) with orthogonal matrices is known to alleviate vanishing and exploding gradient problems. A major objection against such initialisation schemes is that they are deemed biologically implausible as they mandate factorization techniques that are difficult to attribute to a neurobiological process. This paper presents two initialisation schemes that allow a network to naturally evolve its weights to form orthogonal matrices, provides theoretical analysis that pre-training orthogonalisation always converges, and empirically confirms that the proposed schemes outperform randomly initialised recurrent and feedforward networks.
arXiv comment9 pages, 3 figures, to be published in ISCMI2022 conference proceedings
References in corpus (8)
- On the difficulty of training Recurrent Neural Networks
- The Loss Surfaces of Multilayer Networks
- A Simple Way to Initialize Recurrent Networks of Rectified Linear Units
- Unitary Evolution Recurrent Neural Networks
- On orthogonality and learning recurrent networks with long term dependencies
- Recurrent Orthogonal Networks and Long-Memory Tasks
- Feedforward Initialization for Fast Inference of Deep Generative Networks is biologically plausible
- Gradient descent in Gaussian random fields as a toy model for high-dimensional optimisation in deep learning