Saddle Hierarchy in Dense Associative Memory

Abstract

Dense Associative Memory (DAM) models have been attracting renewed attention since they were shown to be robust to adversarial examples and closely related to cutting edge machine learning paradigms, such as the attention mechanism and generative diffusion. We study a DAM built upon a three-layer Boltzmann machine with Potts hidden units, which represent data clusters and classes. Through a statistical mechanics analysis, we derive saddle-point equations that characterize both the stationary points of DAMs trained on real data and the fixed points of DAMs trained on synthetic data within a teacher-student framework. Based on these results, we propose a novel regularization scheme that makes training significantly more stable. Moreover, we show empirically that our DAM learns interpretable solutions to both supervised and unsupervised classification problems. Pushing our theoretical analysis further, we find that the weights learned by relatively small DAMs correspond to unstable saddle points in larger DAMs. We implement a network-growing algorithm that leverages this saddle-point hierarchy to drastically reduce the computational cost of training dense associative memory.

55 pages, 11 figures

References in corpus (18)

• Statistical physics of inference: Thresholds and algorithms
• Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks
• Phase Diagram of Restricted Boltzmann Machines and Generalised Hopfield Networks with Arbitrary Priors
• Dense Associative Memory is Robust to Adversarial Inputs
• Phase transitions in Restricted Boltzmann Machines with generic priors
• Thermodynamics of Restricted Boltzmann Machines and related learning dynamics
• Spectral Dynamics of Learning Restricted Boltzmann Machines
• The Exponential Capacity of Dense Associative Memories
• Manifold Optimization Over the Set of Doubly Stochastic Matrices: A Second-Order Geometry
• Statistical mechanics of unsupervised feature learning in a restricted Boltzmann machine with binary synapses
• Unsupervised feature learning from finite data by message passing: discontinuous versus continuous phase transition
• Minimal model of permutation symmetry in unsupervised learning
• Hopfield model with planted patterns: a teacher-student self-supervised learning model
• Dense Hopfield Networks in the Teacher-Student Setting
• Gaussian-Spherical Restricted Boltzmann Machines
• The effect of priors on Learning with Restricted Boltzmann Machines
• Modeling Structured Data Learning with Restricted Boltzmann Machines in the Teacher-Student Setting
• Role of zero synapses in unsupervised feature learning