Hierarchical Associative Memory
arXiv:2107.06446
Abstract
Dense Associative Memories or Modern Hopfield Networks have many appealing properties of associative memory. They can do pattern completion, store a large number of memories, and can be described using a recurrent neural network with a degree of biological plausibility and rich feedback between the neurons. At the same time, up until now all the models of this class have had only one hidden layer, and have only been formulated with densely connected network architectures, two aspects that hinder their machine learning applications. This paper tackles this gap and describes a fully recurrent model of associative memory with an arbitrary large number of layers, some of which can be locally connected (convolutional), and a corresponding energy function that decreases on the dynamical trajectory of the neurons' activations. The memories of the full network are dynamically "assembled" using primitives encoded in the synaptic weights of the lower layers, with the "assembling rules" encoded in the synaptic weights of the higher layers. In addition to the bottom-up propagation of information, typical of commonly used feedforward neural networks, the model described has rich top-down feedback from higher layers that help the lower-layer neurons to decide on their response to the input stimuli.
References in corpus (10)
- MLP-Mixer: An all-MLP Architecture for Vision
- A guide to convolution arithmetic for deep learning
- Neural Ordinary Differential Equations
- Do You Even Need Attention? A Stack of Feed-Forward Layers Does Surprisingly Well on ImageNet
- Hopfield Networks is All You Need
- Large Associative Memory Problem in Neurobiology and Machine Learning
- Meta-Learning Deep Energy-Based Memory Models
- Convolutional Bipartite Attractor Networks
- COVID-19 Pneumonia Severity Prediction using Hybrid Convolution-Attention Neural Architectures
- A remark on a paper of Krotov and Hopfield [arXiv:2008.06996]