Variational Dynamic Mixtures
arXiv:2010.10403
Abstract
Deep probabilistic time series forecasting models have become an integral part of machine learning. While several powerful generative models have been proposed, we provide evidence that their associated inference models are oftentimes too limited and cause the generative model to predict mode-averaged dynamics. Modeaveraging is problematic since many real-world sequences are highly multi-modal, and their averaged dynamics are unphysical (e.g., predicted taxi trajectories might run through buildings on the street map). To better capture multi-modality, we develop variational dynamic mixtures (VDM): a new variational family to infer sequential latent variables. The VDM approximate posterior at each time step is a mixture density network, whose parameters come from propagating multiple samples through a recurrent architecture. This results in an expressive multi-modal posterior approximation. In an empirical study, we show that VDM outperforms competing approaches on highly multi-modal datasets from different domains.
References in corpus (18)
- Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
- Conditional Generative Adversarial Nets
- A Recurrent Latent Variable Model for Sequential Data
- Professor Forcing: A New Algorithm for Training Recurrent Networks
- GRU-ODE-Bayes: Continuous modeling of sporadically-observed time series
- State-space models' dirty little secrets: even simple linear Gaussian models can have estimation problems
- A Disentangled Recognition and Nonlinear Dynamics Model for Unsupervised Learning
- Auto-Encoding Sequential Monte Carlo
- Flow-GAN: Combining Maximum Likelihood and Adversarial Learning in Generative Models
- Conditional Flow Variational Autoencoders for Structured Sequence Prediction
- Latent Normalizing Flows for Discrete Sequences
- Generative Temporal Models with Memory
- Tree-Structured Recurrent Switching Linear Dynamical Systems for Multi-Scale Modeling
- Switching Linear Dynamics for Variational Bayes Filtering
- State Space LSTM Models with Particle MCMC Inference
- Recurrent Kalman Networks: Factorized Inference in High-Dimensional Deep Feature Spaces
- Scalable Bayesian Learning for State Space Models using Variational Inference with SMC Samplers
- Adaptive Density Estimation for Generative Models