Unsupervised End-to-End Training with a Self-Defined Target

Abstract

Designing algorithms for versatile AI hardware that can learn on the edge using both labeled and unlabeled data is challenging. Deep end-to-end training methods incorporating phases of self-supervised and supervised learning are accurate and adaptable to input data but self-supervised learning requires even more computational and memory resources than supervised learning, too high for current embedded hardware. Conversely, unsupervised layer-by-layer training, such as Hebbian learning, is more compatible with existing hardware but does not integrate well with supervised learning. To address this, we propose a method enabling networks or hardware designed for end-to-end supervised learning to also perform high-performance unsupervised learning by adding two simple elements to the output layer: Winner-Take-All (WTA) selectivity and homeostasis regularization. These mechanisms introduce a "self-defined target" for unlabeled data, allowing purely unsupervised training for both fully-connected and convolutional layers using backpropagation or equilibrium propagation on datasets like MNIST (up to 99.2%), Fashion-MNIST (up to 90.3%), and SVHN (up to 81.5%). We extend this method to semi-supervised learning, adjusting targets based on data type, achieving 96.6% accuracy with only 600 labeled MNIST samples in a multi-layer perceptron. Our results show that this approach can effectively enable networks and hardware initially dedicated to supervised learning to also perform unsupervised learning, adapting to varying availability of labeled data.

References in corpus (23)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction
• Improving neural networks by preventing co-adaptation of feature detectors
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Temporal Ensembling for Semi-Supervised Learning
• Barlow Twins: Self-Supervised Learning via Redundancy Reduction
• VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning
• First-spike based visual categorization using reward-modulated STDP
• Unsupervised Learning by Competing Hidden Units
• A Cookbook of Self-Supervised Learning
• Demonstration of Decentralized, Physics-Driven Learning
• Training End-to-End Analog Neural Networks with Equilibrium Propagation
• SoftHebb: Bayesian Inference in Unsupervised Hebbian Soft Winner-Take-All Networks
• Equilibrium Propagation with Continual Weight Updates
• Hebbian Deep Learning Without Feedback
• Energy-based learning algorithms for analog computing: a comparative study
• Holomorphic Equilibrium Propagation Computes Exact Gradients Through Finite Size Oscillations
• Semi-supervised learning combining backpropagation and STDP: STDP enhances learning by backpropagation with a small amount of labeled data in a spiking neural network
• Activation Learning by Local Competitions
• Blockwise Self-Supervised Learning at Scale
• Is Bio-Inspired Learning Better than Backprop? Benchmarking Bio Learning vs. Backprop
• Synaptic Plasticity Models and Bio-Inspired Unsupervised Deep Learning: A Survey
• A Pseudo-labelling Auto-Encoder for unsupervised image classification

Cited by in corpus (1)

• Self-Contrastive Forward-Forward Algorithm