Assessing the Scalability of Biologically-Motivated Deep Learning Algorithms and Architectures

Abstract

The backpropagation of error algorithm (BP) is impossible to implement in a real brain. The recent success of deep networks in machine learning and AI, however, has inspired proposals for understanding how the brain might learn across multiple layers, and hence how it might approximate BP. As of yet, none of these proposals have been rigorously evaluated on tasks where BP-guided deep learning has proved critical, or in architectures more structured than simple fully-connected networks. Here we present results on scaling up biologically motivated models of deep learning on datasets which need deep networks with appropriate architectures to achieve good performance. We present results on the MNIST, CIFAR-10, and ImageNet datasets and explore variants of target-propagation (TP) and feedback alignment (FA) algorithms, and explore performance in both fully- and locally-connected architectures. We also introduce weight-transport-free variants of difference target propagation (DTP) modified to remove backpropagation from the penultimate layer. Many of these algorithms perform well for MNIST, but for CIFAR and ImageNet we find that TP and FA variants perform significantly worse than BP, especially for networks composed of locally connected units, opening questions about whether new architectures and algorithms are required to scale these approaches. Our results and implementation details help establish baselines for biologically motivated deep learning schemes going forward.

arXiv commentNIPS 2018. Version 2 contains more experimental data including best hyperparameters found

Cited by in corpus (40)

• Convolutional Neural Networks as a Model of the Visual System: Past, Present, and Future
• Synaptic Plasticity Dynamics for Deep Continuous Local Learning (DECOLLE)
• Stand-Alone Self-Attention in Vision Models
• Learning without feedback: Fixed random learning signals allow for feedforward training of deep neural networks
• Deep Learning without Weight Transport
• Supervised learning in a mechanical system
• Finite Versus Infinite Neural Networks: an Empirical Study
• A Theoretical Framework for Target Propagation
• Principled Training of Neural Networks with Direct Feedback Alignment
• Revisiting Locally Supervised Learning: an Alternative to End-to-end Training
• Loss Landscapes of Regularized Linear Autoencoders
• A Brain-inspired Algorithm for Training Highly Sparse Neural Networks
• Kernelized information bottleneck leads to biologically plausible 3-factor Hebbian learning in deep networks
• SpikeGrad: An ANN-equivalent Computation Model for Implementing Backpropagation with Spikes
• Fully Decoupled Neural Network Learning Using Delayed Gradients
• Spike-based causal inference for weight alignment
• Revisiting Spatial Invariance with Low-Rank Local Connectivity
• Towards Biologically Plausible Convolutional Networks
• Biological credit assignment through dynamic inversion of feedforward networks
• Identifying Learning Rules From Neural Network Observables
• Learning to Learn with Feedback and Local Plasticity
• Direct Feedback Alignment Scales to Modern Deep Learning Tasks and Architectures
• Differentially Private Deep Learning with Direct Feedback Alignment
• Large-Scale Gradient-Free Deep Learning with Recursive Local Representation Alignment
• Convergence and Alignment of Gradient Descent with Random Backpropagation Weights
• Attention Aware Cost Volume Pyramid Based Multi-view Stereo Network for 3D Reconstruction
• Credit Assignment Through Broadcasting a Global Error Vector
• Scaling of hardware-compatible perturbative training algorithms
• Accelerated CNN Training Through Gradient Approximation
• Equilibrium Propagation for Complete Directed Neural Networks
• Training DNNs in O(1) memory with MEM-DFA using Random Matrices
• Biologically-Motivated Deep Learning Method using Hierarchical Competitive Learning
• Back to Square One: Superhuman Performance in Chutes and Ladders Through Deep Neural Networks and Tree Search
• Towards On-Chip Bayesian Neuromorphic Learning
• Evolution of diverse (and advanced) cognitive abilities through adaptive fine-tuning of learning and chunking mechanisms
• Front Contribution instead of Back Propagation
• Convergence Analysis and Implicit Regularization of Feedback Alignment for Deep Linear Networks
• Are skip connections necessary for biologically plausible learning rules?
• Deep learning with asymmetric connections and Hebbian updates
• How and When Random Feedback Works: A Case Study of Low-Rank Matrix Factorization