Biologically-plausible learning algorithms can scale to large datasets

Abstract

The backpropagation (BP) algorithm is often thought to be biologically implausible in the brain. One of the main reasons is that BP requires symmetric weight matrices in the feedforward and feedback pathways. To address this "weight transport problem" (Grossberg, 1987), two more biologically plausible algorithms, proposed by Liao et al. (2016) and Lillicrap et al. (2016), relax BP's weight symmetry requirements and demonstrate comparable learning capabilities to that of BP on small datasets. However, a recent study by Bartunov et al. (2018) evaluate variants of target-propagation (TP) and feedback alignment (FA) on MINIST, CIFAR, and ImageNet datasets, and find that although many of the proposed algorithms perform well on MNIST and CIFAR, they perform significantly worse than BP on ImageNet. Here, we additionally evaluate the sign-symmetry algorithm (Liao et al., 2016), which differs from both BP and FA in that the feedback and feedforward weights share signs but not magnitudes. We examine the performance of sign-symmetry and feedback alignment on ImageNet and MS COCO datasets using different network architectures (ResNet-18 and AlexNet for ImageNet, RetinaNet for MS COCO). Surprisingly, networks trained with sign-symmetry can attain classification performance approaching that of BP-trained networks. These results complement the study by Bartunov et al. (2018), and establish a new benchmark for future biologically plausible learning algorithms on more difficult datasets and more complex architectures.

References in corpus (3)

• Focal Loss for Dense Object Detection
• One weird trick for parallelizing convolutional neural networks
• How Important is Weight Symmetry in Backpropagation?

Cited by in corpus (15)

• Putting An End to End-to-End: Gradient-Isolated Learning of Representations
• Deep Learning without Weight Transport
• Feedback alignment in deep convolutional networks
• A Theoretical Framework for Target Propagation
• Learning to solve the credit assignment problem
• Kernelized information bottleneck leads to biologically plausible 3-factor Hebbian learning in deep networks
• Biological credit assignment through dynamic inversion of feedforward networks
• Identifying Learning Rules From Neural Network Observables
• Large-Scale Gradient-Free Deep Learning with Recursive Local Representation Alignment
• Direct Feedback Alignment Scales to Modern Deep Learning Tasks and Architectures
• Accelerated CNN Training Through Gradient Approximation
• Meta-Learning Bidirectional Update Rules
• Benchmarking the Accuracy and Robustness of Feedback Alignment Algorithms
• Tightening the Biological Constraints on Gradient-Based Predictive Coding
• A More Biologically Plausible Local Learning Rule for ANNs