Product Kanerva Machines: Factorized Bayesian Memory
arXiv:2002.02385
Abstract
An ideal cognitively-inspired memory system would compress and organize incoming items. The Kanerva Machine (Wu et al, 2018) is a Bayesian model that naturally implements online memory compression. However, the organization of the Kanerva Machine is limited by its use of a single Gaussian random matrix for storage. Here we introduce the Product Kanerva Machine, which dynamically combines many smaller Kanerva Machines. Its hierarchical structure provides a principled way to abstract invariant features and gives scaling and capacity advantages over single Kanerva Machines. We show that it can exhibit unsupervised clustering, find sparse and combinatorial allocation patterns, and discover spatial tunings that approximately factorize simple images by object.
References in corpus (19)
- Adam: A Method for Stochastic Optimization
- Categorical Reparameterization with Gumbel-Softmax
- End-To-End Memory Networks
- Towards a Definition of Disentangled Representations
- MONet: Unsupervised Scene Decomposition and Representation
- Generative replay with feedback connections as a general strategy for continual learning
- Unsupervised Predictive Memory in a Goal-Directed Agent
- DARLA: Improving Zero-Shot Transfer in Reinforcement Learning
- Generalized Product of Experts for Automatic and Principled Fusion of Gaussian Process Predictions
- Neural probabilistic motor primitives for humanoid control
- Recurrent Independent Mechanisms
- Spatial Broadcast Decoder: A Simple Architecture for Learning Disentangled Representations in VAEs
- Generalization of Reinforcement Learners with Working and Episodic Memory
- Meta-Learning Deep Energy-Based Memory Models
- Online Learning with Gated Linear Networks
- CompILE: Compositional Imitation Learning and Execution
- The Kanerva Machine: A Generative Distributed Memory
- Learning Attractor Dynamics for Generative Memory
- MEMO: A Deep Network for Flexible Combination of Episodic Memories
Cited by in corpus (5)
- Attention Approximates Sparse Distributed Memory
- Gaussian Gated Linear Networks
- Sparse Meta Networks for Sequential Adaptation and its Application to Adaptive Language Modelling
- Meta-Learning with Variational Bayes
- Kanerva++: extending The Kanerva Machine with differentiable, locally block allocated latent memory