Kernelized information bottleneck leads to biologically plausible 3-factor Hebbian learning in deep networks

Abstract

The state-of-the art machine learning approach to training deep neural networks, backpropagation, is implausible for real neural networks: neurons need to know their outgoing weights; training alternates between a bottom-up forward pass (computation) and a top-down backward pass (learning); and the algorithm often needs precise labels of many data points. Biologically plausible approximations to backpropagation, such as feedback alignment, solve the weight transport problem, but not the other two. Thus, fully biologically plausible learning rules have so far remained elusive. Here we present a family of learning rules that does not suffer from any of these problems. It is motivated by the information bottleneck principle (extended with kernel methods), in which networks learn to compress the input as much as possible without sacrificing prediction of the output. The resulting rules have a 3-factor Hebbian structure: they require pre- and post-synaptic firing rates and an error signal - the third factor - consisting of a global teaching signal and a layer-specific term, both available without a top-down pass. They do not require precise labels; instead, they rely on the similarity between pairs of desired outputs. Moreover, to obtain good performance on hard problems and retain biological plausibility, our rules need divisive normalization - a known feature of biological networks. Finally, simulations show that our rules perform nearly as well as backpropagation on image classification tasks.

arXiv commentAccepted to NeurIPS 2020

References in corpus (22)

• Adam: A Method for Stochastic Optimization
• Decoupled Weight Decay Regularization
• A Simple Framework for Contrastive Learning of Visual Representations
• Representation Learning with Contrastive Predictive Coding
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• The information bottleneck method
• Opening the Black Box of Deep Neural Networks via Information
• Deep Variational Information Bottleneck
• Direct Feedback Alignment Provides Learning in Deep Neural Networks
• Unsupervised Learning by Competing Hidden Units
• Training Neural Networks with Local Error Signals
• How Auto-Encoders Could Provide Credit Assignment in Deep Networks via Target Propagation
• Biologically plausible deep learning -- but how far can we go with shallow networks?
• Assessing the Scalability of Biologically-Motivated Deep Learning Algorithms and Architectures
• Putting An End to End-to-End: Gradient-Isolated Learning of Representations
• Deep Learning without Weight Transport
• Why do similarity matching objectives lead to Hebbian/anti-Hebbian networks?
• Normalizing the Normalizers: Comparing and Extending Network Normalization Schemes
• Feedback alignment in deep convolutional networks
• Biologically-plausible learning algorithms can scale to large datasets
• Online Learning with Gated Linear Networks
• Contrastive Similarity Matching for Supervised Learning

Cited by in corpus (5)

• SoftHebb: Bayesian Inference in Unsupervised Hebbian Soft Winner-Take-All Networks
• Self-Supervised Learning with Kernel Dependence Maximization
• Towards Biologically Plausible Convolutional Networks
• Credit Assignment Through Broadcasting a Global Error Vector
• Neuromodulated Neural Architectures with Local Error Signals for Memory-Constrained Online Continual Learning