Learning to Generate with Memory

Abstract

Memory units have been widely used to enrich the capabilities of deep networks on capturing long-term dependencies in reasoning and prediction tasks, but little investigation exists on deep generative models (DGMs) which are good at inferring high-level invariant representations from unlabeled data. This paper presents a deep generative model with a possibly large external memory and an attention mechanism to capture the local detail information that is often lost in the bottom-up abstraction process in representation learning. By adopting a smooth attention model, the whole network is trained end-to-end by optimizing a variational bound of data likelihood via auto-encoding variational Bayesian methods, where an asymmetric recognition network is learnt jointly to infer high-level invariant representations. The asymmetric architecture can reduce the competition between bottom-up invariant feature extraction and top-down generation of instance details. Our experiments on several datasets demonstrate that memory can significantly boost the performance of DGMs and even achieve state-of-the-art results on various tasks, including density estimation, image generation, and missing value imputation.

References in corpus (10)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks
• Semi-Supervised Learning with Deep Generative Models
• Theano: new features and speed improvements
• Recurrent Models of Visual Attention
• Semi-Supervised Learning with Ladder Networks
• Learning to Transduce with Unbounded Memory
• Max-margin Deep Generative Models
• Learning Deep Generative Models with Doubly Stochastic MCMC

Cited by in corpus (5)

• Memorizing Normality to Detect Anomaly: Memory-augmented Deep Autoencoder for Unsupervised Anomaly Detection
• Learning to Learn Variational Semantic Memory
• Progressive Learning and Disentanglement of Hierarchical Representations
• SimulLR: Simultaneous Lip Reading Transducer with Attention-Guided Adaptive Memory
• NeurInt : Learning to Interpolate through Neural ODEs