Unsupervised Learning by Competing Hidden Units

Abstract

It is widely believed that the backpropagation algorithm is essential for learning good feature detectors in early layers of artificial neural networks, so that these detectors are useful for the task performed by the higher layers of that neural network. At the same time, the traditional form of backpropagation is biologically implausible. In the present paper we propose an unusual learning rule, which has a degree of biological plausibility, and which is motivated by Hebb's idea that change of the synapse strength should be local - i.e. should depend only on the activities of the pre and post synaptic neurons. We design a learning algorithm that utilizes global inhibition in the hidden layer, and is capable of learning early feature detectors in a completely unsupervised way. These learned lower layer feature detectors can be used to train higher layer weights in a usual supervised way so that the performance of the full network is comparable to the performance of standard feedforward networks trained end-to-end with a backpropagation algorithm.

References in corpus (10)

• Deep Learning using Linear Support Vector Machines
• Direct Feedback Alignment Provides Learning in Deep Neural Networks
• Random feedback weights support learning in deep neural networks
• A Hebbian/Anti-Hebbian Neural Network for Linear Subspace Learning: A Derivation from Multidimensional Scaling of Streaming Data
• Dense Associative Memory is Robust to Adversarial Inputs
• Why do similarity matching objectives lead to Hebbian/anti-Hebbian networks?
• A Hebbian/Anti-Hebbian Network Derived from Online Non-Negative Matrix Factorization Can Cluster and Discover Sparse Features
• Dendritic error backpropagation in deep cortical microcircuits
• Conducting Credit Assignment by Aligning Local Representations
• A correlation game for unsupervised learning yields computational interpretations of Hebbian excitation, anti-Hebbian inhibition, and synapse elimination

Cited by in corpus (36)

• Biologically plausible deep learning -- but how far can we go with shallow networks?
• Deep Learning without Weight Transport
• On Interpretability of Artificial Neural Networks: A Survey
• Associative Memories via Predictive Coding
• SoftHebb: Bayesian Inference in Unsupervised Hebbian Soft Winner-Take-All Networks
• Dynamical phase transitions in the nonreciprocal Ising model
• Hebbian Semi-Supervised Learning in a Sample Efficiency Setting
• Kernelized information bottleneck leads to biologically plausible 3-factor Hebbian learning in deep networks
• eBrainII: A 3 kW Realtime Custom 3D DRAM integrated ASIC implementation of a Biologically Plausible Model of a Human Scale Cortex
• Efficiency of local learning rules in threshold-linear associative networks
• Learning representations in Bayesian Confidence Propagation neural networks
• Prototype-based interpretation of the functionality of neurons in winner-take-all neural networks
• Physical Neural Networks with Self-Learning Capabilities
• Self-Contrastive Forward-Forward Algorithm
• Parallel Training of Deep Networks with Local Updates
• Local Unsupervised Learning for Image Analysis
• Bio-Inspired Hashing for Unsupervised Similarity Search
• CSNNs: Unsupervised, Backpropagation-free Convolutional Neural Networks for Representation Learning
• Spiking Neural Predictive Coding for Continual Learning from Data Streams
• Large-Scale Gradient-Free Deep Learning with Recursive Local Representation Alignment
• Advancing the Biological Plausibility and Efficacy of Hebbian Convolutional Neural Networks
• Attack and defence in cellular decision-making: lessons from machine learning
• PyTorch-Hebbian: facilitating local learning in a deep learning framework
• Credit Assignment Through Broadcasting a Global Error Vector
• Structured and Deep Similarity Matching via Structured and Deep Hebbian Networks
• Representing Prior Knowledge Using Randomly, Weighted Feature Networks for Visual Relationship Detection
• Unsupervised Hebbian Learning on Point Sets in StarCraft II
• Unsupervised End-to-End Training with a Self-Defined Target
• Biologically-Motivated Deep Learning Method using Hierarchical Competitive Learning
• A feedback control optimizer for online and hardware-aware training of Spiking Neural Networks
• Exploring Biologically Inspired Mechanisms of Adversarial Robustness
• Network Dynamics Governed by Lyapunov Functions: From Memory to Classification
• A Biologically Plausible Learning Rule for Perceptual Systems of organisms that Maximize Mutual Information
• Neuromodulation via Krotov-Hopfield Improves Accuracy and Robustness of RBMs
• Hebbian Learning with Global Direction
• Can a Fruit Fly Learn Word Embeddings?