Biologically plausible deep learning -- but how far can we go with shallow networks?

Abstract

Training deep neural networks with the error backpropagation algorithm is considered implausible from a biological perspective. Numerous recent publications suggest elaborate models for biologically plausible variants of deep learning, typically defining success as reaching around 98% test accuracy on the MNIST data set. Here, we investigate how far we can go on digit (MNIST) and object (CIFAR10) classification with biologically plausible, local learning rules in a network with one hidden layer and a single readout layer. The hidden layer weights are either fixed (random or random Gabor filters) or trained with unsupervised methods (PCA, ICA or Sparse Coding) that can be implemented by local learning rules. The readout layer is trained with a supervised, local learning rule. We first implement these models with rate neurons. This comparison reveals, first, that unsupervised learning does not lead to better performance than fixed random projections or Gabor filters for large hidden layers. Second, networks with localized receptive fields perform significantly better than networks with all-to-all connectivity and can reach backpropagation performance on MNIST. We then implement two of the networks - fixed, localized, random & random Gabor filters in the hidden layer - with spiking leaky integrate-and-fire neurons and spike timing dependent plasticity to train the readout layer. These spiking models achieve > 98.2% test accuracy on MNIST, which is close to the performance of rate networks with one hidden layer trained with backpropagation. The performance of our shallow network models is comparable to most current biologically plausible models of deep learning. Furthermore, our results with a shallow spiking network provide an important reference and suggest the use of datasets other than MNIST for testing the performance of future models of biologically plausible deep learning.

arXiv comment14 pages, 4 figures

References in corpus (18)

• Deep Learning in Neural Networks: An Overview
• Deep Learning in Spiking Neural Networks
• STDP-based spiking deep convolutional neural networks for object recognition
• Neuromorphic Deep Learning Machines
• A sparse coding model with synaptically local plasticity and spiking neurons can account for the diverse shapes of V1 simple cell receptive fields
• Event-Driven Contrastive Divergence for Spiking Neuromorphic Systems
• Direct Feedback Alignment Provides Learning in Deep Neural Networks
• Unsupervised Learning by Competing Hidden Units
• Nonlinear Hebbian learning as a unifying principle in receptive field formation
• Bio-Inspired Spiking Convolutional Neural Network using Layer-wise Sparse Coding and STDP Learning
• Feedback alignment in deep convolutional networks
• A Hebbian/Anti-Hebbian Network Derived from Online Non-Negative Matrix Factorization Can Cluster and Discover Sparse Features
• Dendritic error backpropagation in deep cortical microcircuits
• How far can we go without convolution: Improving fully-connected networks
• A Biologically Plausible Learning Rule for Deep Learning in the Brain
• Error Forward-Propagation: Reusing Feedforward Connections to Propagate Errors in Deep Learning
• Temporally Efficient Deep Learning with Spikes
• Unsupervised Regenerative Learning of Hierarchical Features in Spiking Deep Networks for Object Recognition

Cited by in corpus (16)

• S4NN: temporal backpropagation for spiking neural networks with one spike per neuron
• BS4NN: Binarized Spiking Neural Networks with Temporal Coding and Learning
• Drawing Inspiration from Biological Dendrites to Empower Artificial Neural Networks
• SoftHebb: Bayesian Inference in Unsupervised Hebbian Soft Winner-Take-All Networks
• Kernelized information bottleneck leads to biologically plausible 3-factor Hebbian learning in deep networks
• Predictive Coding Can Do Exact Backpropagation on Convolutional and Recurrent Neural Networks
• Learning representations in Bayesian Confidence Propagation neural networks
• Fast and energy-efficient neuromorphic deep learning with first-spike times
• A differential Hebbian framework for biologically-plausible motor control
• Training Deep Spiking Auto-encoders without Bursting or Dying Neurons through Regularization
• Binary autoencoder with random binary weights
• Evaluating adversarial robustness in simulated cerebellum
• 2022 Roadmap on Neuromorphic Computing and Engineering
• BioLeaF: A Bio-plausible Learning Framework for Training of Spiking Neural Networks
• Hebbian Learning with Global Direction
• A Computational Model of Representation Learning in the Brain Cortex, Integrating Unsupervised and Reinforcement Learning