Robust High-dimensional Memory-augmented Neural Networks

Abstract

Traditional neural networks require enormous amounts of data to build their complex mappings during a slow training procedure that hinders their abilities for relearning and adapting to new data. Memory-augmented neural networks enhance neural networks with an explicit memory to overcome these issues. Access to this explicit memory, however, occurs via soft read and write operations involving every individual memory entry, resulting in a bottleneck when implemented using the conventional von Neumann computer architecture. To overcome this bottleneck, we propose a robust architecture that employs a computational memory unit as the explicit memory performing analog in-memory computation on high-dimensional (HD) vectors, while closely matching 32-bit software-equivalent accuracy. This is achieved by a content-based attention mechanism that represents unrelated items in the computational memory with uncorrelated HD vectors, whose real-valued components can be readily approximated by binary, or bipolar components. Experimental results demonstrate the efficacy of our approach on few-shot image classification tasks on the Omniglot dataset using more than 256,000 phase-change memory devices. Our approach effectively merges the richness of deep neural network representations with HD computing that paves the way for robust vector-symbolic manipulations applicable in reasoning, fusion, and compression.

This is a pre-print of an article accepted for publication in Nature Communications

References in corpus (10)

• TensorFlow: A system for large-scale machine learning
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Prototypical Networks for Few-shot Learning
• Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
• Matching Networks for One Shot Learning
• Accurate deep neural network inference using computational phase-change memory
• One-shot Learning with Memory-Augmented Neural Networks
• Vector Symbolic Architectures answer Jackendoff's challenges for cognitive neuroscience
• Few-Shot Learning with Global Class Representations
• The Kanerva Machine: A Generative Distributed Memory

Cited by in corpus (10)

• A Survey on Hyperdimensional Computing aka Vector Symbolic Architectures, Part II: Applications, Cognitive Models, and Challenges
• A Survey on Hyperdimensional Computing aka Vector Symbolic Architectures, Part I: Models and Data Transformations
• A comparison of Vector Symbolic Architectures
• A TinyML Platform for On-Device Continual Learning with Quantized Latent Replays
• Vector Symbolic Architectures as a Computing Framework for Emerging Hardware
• HW/SW Co-design for Reliable TCAM-based In-memory Brain-inspired Hyperdimensional Computing
• Spiking Hyperdimensional Network: Neuromorphic Models Integrated with Memory-Inspired Framework
• Generalized Key-Value Memory to Flexibly Adjust Redundancy in Memory-Augmented Networks
• WHYPE: A Scale-Out Architecture with Wireless Over-the-Air Majority for Scalable In-memory Hyperdimensional Computing
• Integer Factorization with Compositional Distributed Representations