Recurrent Gaussian Processes

Abstract

We define Recurrent Gaussian Processes (RGP) models, a general family of Bayesian nonparametric models with recurrent GP priors which are able to learn dynamical patterns from sequential data. Similar to Recurrent Neural Networks (RNNs), RGPs can have different formulations for their internal states, distinct inference methods and be extended with deep structures. In such context, we propose a novel deep RGP model whose autoregressive states are latent, thereby performing representation and dynamical learning simultaneously. To fully exploit the Bayesian nature of the RGP model we develop the Recurrent Variational Bayes (REVARB) framework, which enables efficient inference and strong regularization through coherent propagation of uncertainty across the RGP layers and states. We also introduce a RGP extension where variational parameters are greatly reduced by being reparametrized through RNN-based sequential recognition models. We apply our model to the tasks of nonlinear system identification and human motion modeling. The promising obtained results indicate that our RGP model maintains its highly flexibility while being able to avoid overfitting and being applicable even when larger datasets are not available.

Published as a conference paper at ICLR 2016. 12 pages, 3 figures

References in corpus (4)

• Modeling Temporal Dependencies in High-Dimensional Sequences: Application to Polyphonic Music Generation and Transcription
• Semi-described and semi-supervised learning with Gaussian processes
• Computationally Efficient Bayesian Learning of Gaussian Process State Space Models
• Note on Equivalence Between Recurrent Neural Network Time Series Models and Variational Bayesian Models

Cited by in corpus (15)

• How priors of initial hyperparameters affect Gaussian process regression models
• Few-Shot Deep Adversarial Learning for Video-based Person Re-identification
• When Gaussian Process Meets Big Data: A Review of Scalable GPs
• The Functional Neural Process
• Recurrent Attentive Neural Process for Sequential Data
• Neural Dynamics Discovery via Gaussian Process Recurrent Neural Networks
• Streaming Variational Monte Carlo
• Identification of Gaussian Process State Space Models
• Probabilistic Recurrent State-Space Models
• Computationally Efficient Bayesian Learning of Gaussian Process State Space Models
• Structured Variational Inference in Unstable Gaussian Process State Space Models
• A Survey of Techniques All Classifiers Can Learn from Deep Networks: Models, Optimizations, and Regularization
• Neural Likelihoods for Multi-Output Gaussian Processes
• Evolving-Graph Gaussian Processes
• Modeling continuous-time stochastic processes using $\mathcal{N}$-Curve mixtures