Overparameterized Neural Networks Implement Associative Memory

Abstract

Identifying computational mechanisms for memorization and retrieval of data is a long-standing problem at the intersection of machine learning and neuroscience. Our main finding is that standard overparameterized deep neural networks trained using standard optimization methods implement such a mechanism for real-valued data. Empirically, we show that: (1) overparameterized autoencoders store training samples as attractors, and thus, iterating the learned map leads to sample recovery; (2) the same mechanism allows for encoding sequences of examples, and serves as an even more efficient mechanism for memory than autoencoding. Theoretically, we prove that when trained on a single example, autoencoders store the example as an attractor. Lastly, by treating a sequence encoder as a composition of maps, we prove that sequence encoding provides a more efficient mechanism for memory than autoencoding.

References in corpus (14)

• Adam: A Method for Stochastic Optimization
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Pixel Recurrent Neural Networks
• Reconciling modern machine learning practice and the bias-variance trade-off
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Understanding deep learning requires rethinking generalization
• Gradient Descent Provably Optimizes Over-parameterized Neural Networks
• A Closer Look at Memorization in Deep Networks
• Gradient Descent Finds Global Minima of Deep Neural Networks
• On the Optimization of Deep Networks: Implicit Acceleration by Overparameterization
• A Convergence Analysis of Gradient Descent for Deep Linear Neural Networks
• What Regularized Auto-Encoders Learn from the Data Generating Distribution
• Global Convergence of Adaptive Gradient Methods for An Over-parameterized Neural Network
• Meta-Learning Deep Energy-Based Memory Models

Cited by in corpus (5)

• Causal Network Models of SARS-CoV-2 Expression and Aging to Identify Candidates for Drug Repurposing
• Honest-but-Curious Nets: Sensitive Attributes of Private Inputs Can Be Secretly Coded into the Classifiers' Outputs
• Simple, Fast, and Flexible Framework for Matrix Completion with Infinite Width Neural Networks
• Contrastive estimation reveals topic posterior information to linear models
• Learning and Generalization in Overparameterized Normalizing Flows