On the biological plausibility of orthogonal initialisation for solving gradient instability in deep neural networks

Nikolay Manchev, Michael Spratling

Initialising the synaptic weights of artificial neural networks (ANNs) with orthogonal matrices is known to alleviate vanishing and exploding gradient problems. A major objection a…