Synaptic balancing: a biologically plausible local learning rule that provably increases neural network noise robustness without sacrificing task performance

Abstract

We introduce a novel, biologically plausible local learning rule that provably increases the robustness of neural dynamics to noise in nonlinear recurrent neural networks with homogeneous nonlinearities. Our learning rule achieves higher noise robustness without sacrificing performance on the task and without requiring any knowledge of the particular task. The plasticity dynamics -- an integrable dynamical system operating on the weights of the network -- maintains a multiplicity of conserved quantities, most notably the network's entire temporal map of input to output trajectories. The outcome of our learning rule is a synaptic balancing between the incoming and outgoing synapses of every neuron. This synaptic balancing rule is consistent with many known aspects of experimentally observed heterosynaptic plasticity, and moreover makes new experimentally testable predictions relating plasticity at the incoming and outgoing synapses of individual neurons. Overall, this work provides a novel, practical local learning rule that exactly preserves overall network function and, in doing so, provides new conceptual bridges between the disparate worlds of the neurobiology of heterosynaptic plasticity, the engineering of regularized noise-robust networks, and the mathematics of integrable Lax dynamical systems.

References in corpus (5)

• Identifying and attacking the saddle point problem in high-dimensional non-convex optimization
• Pruning neural networks without any data by iteratively conserving synaptic flow
• Universality and individuality in neural dynamics across large populations of recurrent networks
• Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balanced
• Predictive coding in balanced neural networks with noise, chaos and delays