Meta-Learning Deep Energy-Based Memory Models

Abstract

We study the problem of learning associative memory -- a system which is able to retrieve a remembered pattern based on its distorted or incomplete version. Attractor networks provide a sound model of associative memory: patterns are stored as attractors of the network dynamics and associative retrieval is performed by running the dynamics starting from a query pattern until it converges to an attractor. In such models the dynamics are often implemented as an optimization procedure that minimizes an energy function, such as in the classical Hopfield network. In general it is difficult to derive a writing rule for a given dynamics and energy that is both compressive and fast. Thus, most research in energy-based memory has been limited either to tractable energy models not expressive enough to handle complex high-dimensional objects such as natural images, or to models that do not offer fast writing. We present a novel meta-learning approach to energy-based memory models (EBMM) that allows one to use an arbitrary neural architecture as an energy model and quickly store patterns in its weights. We demonstrate experimentally that our EBMM approach can build compressed memories for synthetic and natural data, and is capable of associative retrieval that outperforms existing memory systems in terms of the reconstruction error and compression rate.

arXiv commentICLR 2020

References in corpus (16)

• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Decoupled Weight Decay Regularization
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• Theoretical Models of Learning to Learn
• Understanding deep learning requires rethinking generalization
• A Theory of Generative ConvNet
• Using Fast Weights to Attend to the Recent Past
• Implicit Generation and Generalization in Energy-Based Models
• Recurrent Inference Machines for Solving Inverse Problems
• How to train your MAML
• Few-shot Autoregressive Density Estimation: Towards Learning to Learn Distributions
• Iterative Amortized Inference
• Metalearned Neural Memory
• Learning Attractor Dynamics for Generative Memory
• Fast Adaptation in Generative Models with Generative Matching Networks

Cited by in corpus (10)

• Overparameterized Neural Networks Implement Associative Memory
• Associative Memories via Predictive Coding
• Improved Contrastive Divergence Training of Energy Based Models
• Hierarchical Associative Memory
• Compositional Visual Generation and Inference with Energy Based Models
• Energy-Based Models for Continual Learning
• Product Kanerva Machines: Factorized Bayesian Memory
• NOVAS: Non-convex Optimization via Adaptive Stochastic Search for End-to-End Learning and Control
• Energy-Based Learning for Cooperative Games, with Applications to Valuation Problems in Machine Learning
• Meta-Learning for One-Class Classification with Few Examples using Order-Equivariant Network