Synaptic metaplasticity in binarized neural networks

Abstract

Unlike the brain, artificial neural networks, including state-of-the-art deep neural networks for computer vision, are subject to "catastrophic forgetting": they rapidly forget the previous task when trained on a new one. Neuroscience suggests that biological synapses avoid this issue through the process of synaptic consolidation and metaplasticity: the plasticity itself changes upon repeated synaptic events. In this work, we show that this concept of metaplasticity can be transferred to a particular type of deep neural networks, binarized neural networks, to reduce catastrophic forgetting.

arXiv comment3 pages, 1 figure

References in corpus (6)

• Overcoming catastrophic forgetting in neural networks
• Continual Learning Through Synaptic Intelligence
• Vowel recognition with four coupled spin-torque nano-oscillators
• Network Plasticity as Bayesian Inference
• XNOR Neural Engine: a Hardware Accelerator IP for 21.6 fJ/op Binary Neural Network Inference
• Latent Weights Do Not Exist: Rethinking Binarized Neural Network Optimization

Cited by in corpus (7)

• Continual Learning and Catastrophic Forgetting
• Deep learning via message passing algorithms based on belief propagation
• Eight challenges in developing theory of intelligence
• Statistical mechanics of continual learning: variational principle and mean-field potential
• Synaptic metaplasticity with multi-level memristive devices
• Quantized Non-Volatile Nanomagnetic Synapse based Autoencoder for Efficient Unsupervised Network Anomaly Detection
• Metaplastic and Energy-Efficient Biocompatible Graphene Artificial Synaptic Transistors for Enhanced Accuracy Neuromorphic Computing