Kanerva++: extending The Kanerva Machine with differentiable, locally block allocated latent memory
arXiv:2103.03905
Abstract
Episodic and semantic memory are critical components of the human memory model. The theory of complementary learning systems (McClelland et al., 1995) suggests that the compressed representation produced by a serial event (episodic memory) is later restructured to build a more generalized form of reusable knowledge (semantic memory). In this work we develop a new principled Bayesian memory allocation scheme that bridges the gap between episodic and semantic memory via a hierarchical latent variable model. We take inspiration from traditional heap allocation and extend the idea of locally contiguous memory to the Kanerva Machine, enabling a novel differentiable block allocated latent memory. In contrast to the Kanerva Machine, we simplify the process of memory writing by treating it as a fully feed forward deterministic process, relying on the stochasticity of the read key distribution to disperse information within the memory. We demonstrate that this allocation scheme improves performance in memory conditional image generation, resulting in new state-of-the-art conditional likelihood values on binarized MNIST (<=41.58 nats/image) , binarized Omniglot (<=66.24 nats/image), as well as presenting competitive performance on CIFAR10, DMLab Mazes, Celeb-A and ImageNet32x32.
References in corpus (18)
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- Auto-Encoding Variational Bayes
- Spectral Normalization for Generative Adversarial Networks
- Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
- Variational Graph Auto-Encoders
- Large Batch Training of Convolutional Networks
- Visual Reinforcement Learning with Imagined Goals
- Variational Lossy Autoencoder
- Neural Episodic Control
- PixelVAE: A Latent Variable Model for Natural Images
- Evolving Normalization-Activation Layers
- An Architecture for Deep, Hierarchical Generative Models
- Associative Compression Networks for Representation Learning
- PixelVAE++: Improved PixelVAE with Discrete Prior
- MAE: Mutual Posterior-Divergence Regularization for Variational AutoEncoders
- Variational Memory Encoder-Decoder
- Learning Attractor Dynamics for Generative Memory
- Product Kanerva Machines: Factorized Bayesian Memory