Deep Learning without Weight Transport

Abstract

Current algorithms for deep learning probably cannot run in the brain because they rely on weight transport, where forward-path neurons transmit their synaptic weights to a feedback path, in a way that is likely impossible biologically. An algorithm called feedback alignment achieves deep learning without weight transport by using random feedback weights, but it performs poorly on hard visual-recognition tasks. Here we describe two mechanisms - a neural circuit called a weight mirror and a modification of an algorithm proposed by Kolen and Pollack in 1994 - both of which let the feedback path learn appropriate synaptic weights quickly and accurately even in large networks, without weight transport or complex wiring.Tested on the ImageNet visual-recognition task, these mechanisms outperform both feedback alignment and the newer sign-symmetry method, and nearly match backprop, the standard algorithm of deep learning, which uses weight transport.

arXiv commentAccepted for the Conference on Neural Information Processing Systems (NeurIPS) 2019

References in corpus (8)

• Unsupervised Learning by Competing Hidden Units
• Assessing the Scalability of Biologically-Motivated Deep Learning Algorithms and Architectures
• Feedback alignment in deep convolutional networks
• Biologically-plausible learning algorithms can scale to large datasets
• TF-Replicator: Distributed Machine Learning for Researchers
• Loss Landscapes of Regularized Linear Autoencoders
• Understanding Reuse, Performance, and Hardware Cost of DNN Dataflows: A Data-Centric Approach Using MAESTRO
• Spike-based causal inference for weight alignment

Cited by in corpus (26)

• A Theoretical Framework for Target Propagation
• Learning to solve the credit assignment problem
• Principled Training of Neural Networks with Direct Feedback Alignment
• Predictive Coding: a Theoretical and Experimental Review
• Two Routes to Scalable Credit Assignment without Weight Symmetry
• Loss Landscapes of Regularized Linear Autoencoders
• Kernelized information bottleneck leads to biologically plausible 3-factor Hebbian learning in deep networks
• Relaxing the Constraints on Predictive Coding Models
• GAIT-prop: A biologically plausible learning rule derived from backpropagation of error
• Activation Relaxation: A Local Dynamical Approximation to Backpropagation in the Brain
• Towards Biologically Plausible Convolutional Networks
• Biological credit assignment through dynamic inversion of feedforward networks
• Identifying Learning Rules From Neural Network Observables
• Biologically Plausible Training Mechanisms for Self-Supervised Learning in Deep Networks
• Event-Driven Visual-Tactile Sensing and Learning for Robots
• Differentially Private Deep Learning with Direct Feedback Alignment
• Direct Feedback Alignment Scales to Modern Deep Learning Tasks and Architectures
• Large-Scale Gradient-Free Deep Learning with Recursive Local Representation Alignment
• Convergence and Alignment of Gradient Descent with Random Backpropagation Weights
• Meta-Learning Bidirectional Update Rules
• Benchmarking the Accuracy and Robustness of Feedback Alignment Algorithms
• On the Adversarial Robustness of Neural Networks without Weight Transport
• Tightening the Biological Constraints on Gradient-Based Predictive Coding
• Structured Mean-field Variational Inference and Learning in Winner-take-all Spiking Neural Networks
• How and When Random Feedback Works: A Case Study of Low-Rank Matrix Factorization
• Neko: a Library for Exploring Neuromorphic Learning Rules