Convolutional Channel-wise Competitive Learning for the Forward-Forward Algorithm

Abstract

The Forward-Forward (FF) Algorithm has been recently proposed to alleviate the issues of backpropagation (BP) commonly used to train deep neural networks. However, its current formulation exhibits limitations such as the generation of negative data, slower convergence, and inadequate performance on complex tasks. In this paper, we take the main ideas of FF and improve them by leveraging channel-wise competitive learning in the context of convolutional neural networks for image classification tasks. A layer-wise loss function is introduced that promotes competitive learning and eliminates the need for negative data construction. To enhance both the learning of compositional features and feature space partitioning, a channel-wise feature separator and extractor block is proposed that complements the competitive learning process. Our method outperforms recent FF-based models on image classification tasks, achieving testing errors of 0.58%, 7.69%, 21.89%, and 48.77% on MNIST, Fashion-MNIST, CIFAR-10 and CIFAR-100 respectively. Our approach bridges the performance gap between FF learning and BP methods, indicating the potential of our proposed approach to learn useful representations in a layer-wise modular fashion, enabling more efficient and flexible learning.

arXiv commentTo be published in AAAI 2024, 11 pages, 7 figures

References in corpus (13)

• Adam: A Method for Stochastic Optimization
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• A Simple Framework for Contrastive Learning of Visual Representations
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Towards Deep Learning Models Resistant to Adversarial Attacks
• Evolution Strategies as a Scalable Alternative to Reinforcement Learning
• The Forward-Forward Algorithm: Some Preliminary Investigations
• Learning without feedback: Fixed random learning signals allow for feedforward training of deep neural networks
• SoftHebb: Bayesian Inference in Unsupervised Hebbian Soft Winner-Take-All Networks
• Black-Box Ripper: Copying black-box models using generative evolutionary algorithms
• Hebbian Deep Learning Without Feedback
• The Predictive Forward-Forward Algorithm
• The Cascaded Forward Algorithm for Neural Network Training

Cited by in corpus (2)

• Self-Contrastive Forward-Forward Algorithm
• Training Convolutional Neural Networks with the Forward-Forward algorithm