Generative Temporal Models with Memory

Abstract

We consider the general problem of modeling temporal data with long-range dependencies, wherein new observations are fully or partially predictable based on temporally-distant, past observations. A sufficiently powerful temporal model should separate predictable elements of the sequence from unpredictable elements, express uncertainty about those unpredictable elements, and rapidly identify novel elements that may help to predict the future. To create such models, we introduce Generative Temporal Models augmented with external memory systems. They are developed within the variational inference framework, which provides both a practical training methodology and methods to gain insight into the models' operation. We show, on a range of problems with sparse, long-term temporal dependencies, that these models store information from early in a sequence, and reuse this stored information efficiently. This allows them to perform substantially better than existing models based on well-known recurrent neural networks, like LSTMs.

References in corpus (12)

• Adam: A Method for Stochastic Optimization
• Auto-Encoding Variational Bayes
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• DRAW: A Recurrent Neural Network For Image Generation
• Ask Me Anything: Dynamic Memory Networks for Natural Language Processing
• One-shot Learning with Memory-Augmented Neural Networks
• Attend, Infer, Repeat: Fast Scene Understanding with Generative Models
• Neural Programmer-Interpreters
• Learning Stochastic Recurrent Networks
• Sequential Neural Models with Stochastic Layers
• Deep Kalman Filters
• Data-Efficient Learning of Feedback Policies from Image Pixels using Deep Dynamical Models

Cited by in corpus (17)

• Unsupervised Predictive Memory in a Goal-Directed Agent
• Small Sample Learning in Big Data Era
• Learning Multimodal Transition Dynamics for Model-Based Reinforcement Learning
• Mastering Atari with Discrete World Models
• Recurrent Attentive Neural Process for Sequential Data
• Generalisation of structural knowledge in the hippocampal-entorhinal system
• Variational Memory Addressing in Generative Models
• Temporal Difference Variational Auto-Encoder
• Clockwork Variational Autoencoders
• The Kanerva Machine: A Generative Distributed Memory
• Exploiting Human Social Cognition for the Detection of Fake and Fraudulent Faces via Memory Networks
• Improving Sequential Latent Variable Models with Autoregressive Flows
• Superposed Episodic and Semantic Memory via Sparse Distributed Representation
• Variational Dynamic Mixtures
• Imitation Learning of Factored Multi-agent Reactive Models
• Memory and attention in deep learning
• A Contemporary Overview of Probabilistic Latent Variable Models