Deep Temporal Sigmoid Belief Networks for Sequence Modeling

Abstract

Deep dynamic generative models are developed to learn sequential dependencies in time-series data. The multi-layered model is designed by constructing a hierarchy of temporal sigmoid belief networks (TSBNs), defined as a sequential stack of sigmoid belief networks (SBNs). Each SBN has a contextual hidden state, inherited from the previous SBNs in the sequence, and is used to regulate its hidden bias. Scalable learning and inference algorithms are derived by introducing a recognition model that yields fast sampling from the variational posterior. This recognition model is trained jointly with the generative model, by maximizing its variational lower bound on the log-likelihood. Experimental results on bouncing balls, polyphonic music, motion capture, and text streams show that the proposed approach achieves state-of-the-art predictive performance, and has the capacity to synthesize various sequences.

to appear in NIPS 2015

References in corpus (7)

• On the difficulty of training Recurrent Neural Networks
• A Recurrent Latent Variable Model for Sequential Data
• Modeling Temporal Dependencies in High-Dimensional Sequences: Application to Polyphonic Music Generation and Transcription
• Learning Stochastic Recurrent Networks
• Variational Recurrent Auto-Encoders
• Nonparametric Bayesian Factor Analysis for Dynamic Count Matrices
• Fast Second-Order Stochastic Backpropagation for Variational Inference

Cited by in corpus (21)

• Dynamical Variational Autoencoders: A Comprehensive Review
• Preconditioned Stochastic Gradient Langevin Dynamics for Deep Neural Networks
• Deep Kalman Filters
• Scalable Gradients for Stochastic Differential Equations
• Black box variational inference for state space models
• ODE${}^2$VAE: Deep generative second order ODEs with Bayesian neural networks
• Unsupervised Learning of Visual Structure using Predictive Generative Networks
• Breaking the Softmax Bottleneck: A High-Rank RNN Language Model
• Variational Temporal Deep Generative Model for Radar HRRP Target Recognition
• Fast Second-Order Stochastic Backpropagation for Variational Inference
• Stochastic WaveNet: A Generative Latent Variable Model for Sequential Data
• Weakly Supervised Deep Recurrent Neural Networks for Basic Dance Step Generation
• Factored Temporal Sigmoid Belief Networks for Sequence Learning
• Improving Sequential Latent Variable Models with Autoregressive Flows
• Learning Partially Known Stochastic Dynamics with Empirical PAC Bayes
• Deep Poisson gamma dynamical systems
• On the difficulty of learning and predicting the long-term dynamics of bouncing objects
• High-Order Stochastic Gradient Thermostats for Bayesian Learning of Deep Models
• Ensemble Kalman Variational Objectives: Nonlinear Latent Trajectory Inference with A Hybrid of Variational Inference and Ensemble Kalman Filter
• Scalable Deep Generative Relational Models with High-Order Node Dependence
• Recurrent Dirichlet Belief Networks for Interpretable Dynamic Relational Data Modelling