Self-Contrastive Forward-Forward Algorithm

Abstract

Agents that operate autonomously benefit from lifelong learning capabilities. However, compatible training algorithms must comply with the decentralized nature of these systems, which imposes constraints on both the parameter counts and the computational resources. The Forward-Forward (FF) algorithm is one of these. FF relies only on feedforward operations, the same used for inference, for optimizing layer-wise objectives. This purely forward approach eliminates the need for transpose operations required in traditional backpropagation. Despite its potential, FF has failed to reach state-of-the-art performance on most standard benchmark tasks, in part due to unreliable negative data generation methods for unsupervised learning. In this work, we propose the Self-Contrastive Forward-Forward (SCFF) algorithm, a competitive training method aimed at closing this performance gap. Inspired by standard self-supervised contrastive learning for vision tasks, SCFF generates positive and negative inputs applicable across various datasets. The method demonstrates superior performance compared to existing unsupervised local learning algorithms on several benchmark datasets, including MNIST, CIFAR-10, STL-10, and Tiny ImageNet. We extend FF's application to training recurrent neural networks, expanding its utility to sequential data tasks. These findings pave the way for high-accuracy, real-time learning on resource-constrained edge devices.

References in corpus (21)

• Neuromorphic Deep Learning Machines
• Unsupervised Learning by Competing Hidden Units
• Silicon Photonic Architecture for Training Deep Neural Networks with Direct Feedback Alignment
• Demonstration of Decentralized, Physics-Driven Learning
• Learning without feedback: Fixed random learning signals allow for feedforward training of deep neural networks
• Robust Inference via Generative Classifiers for Handling Noisy Labels
• Normalization Techniques in Training DNNs: Methodology, Analysis and Application
• Desynchronous Learning in a Physics-Driven Learning Network
• SoftHebb: Bayesian Inference in Unsupervised Hebbian Soft Winner-Take-All Networks
• Hebbian Semi-Supervised Learning in a Sample Efficiency Setting
• Comparing SNNs and RNNs on Neuromorphic Vision Datasets: Similarities and Differences
• Convolutional Channel-wise Competitive Learning for the Forward-Forward Algorithm
• Hardware Beyond Backpropagation: a Photonic Co-Processor for Direct Feedback Alignment
• Self-Contrastive Forward-Forward Algorithm
• Biologically Plausible Training Mechanisms for Self-Supervised Learning in Deep Networks
• Forward Learning with Top-Down Feedback: Empirical and Analytical Characterization
• Role of non-linear data processing on speech recognition task in the framework of reservoir computing
• The Cascaded Forward Algorithm for Neural Network Training
• Backpropagation through space, time, and the brain
• Unsupervised End-to-End Training with a Self-Defined Target
• Distance-Forward Learning: Enhancing the Forward-Forward Algorithm Towards High-Performance On-Chip Learning

Cited by in corpus (1)

• Self-Contrastive Forward-Forward Algorithm