Recurrent Attentive Neural Process for Sequential Data

Abstract

Neural processes (NPs) learn stochastic processes and predict the distribution of target output adaptively conditioned on a context set of observed input-output pairs. Furthermore, Attentive Neural Process (ANP) improved the prediction accuracy of NPs by incorporating attention mechanism among contexts and targets. In a number of real-world applications such as robotics, finance, speech, and biology, it is critical to learn the temporal order and recurrent structure from sequential data. However, the capability of NPs capturing these properties is limited due to its permutation invariance instinct. In this paper, we proposed the Recurrent Attentive Neural Process (RANP), or alternatively, Attentive Neural Process-RecurrentNeural Network(ANP-RNN), in which the ANP is incorporated into a recurrent neural network. The proposed model encapsulates both the inductive biases of recurrent neural networks and also the strength of NPs for modelling uncertainty. We demonstrate that RANP can effectively model sequential data and outperforms NPs and LSTMs remarkably in a 1D regression toy example as well as autonomous-driving applications.

12 pages, 6 figures, NeurIPS 2019 Workshop

References in corpus (14)

• Sequence to Sequence Learning with Neural Networks
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• A Recurrent Latent Variable Model for Sequential Data
• Deep Kalman Filters
• Stochastic Variational Deep Kernel Learning
• Deep Variational Bayes Filters: Unsupervised Learning of State Space Models from Raw Data
• Thoughts on Massively Scalable Gaussian Processes
• Generative Temporal Models with Memory
• Recurrent Gaussian Processes
• Combining LSTM and Latent Topic Modeling for Mortality Prediction
• Learning Scalable Deep Kernels with Recurrent Structure
• The Statistical Recurrent Unit
• Consistent Generative Query Networks
• Probabilistic Recurrent State-Space Models

Cited by in corpus (9)

• Spatiotemporal Learning of Multivehicle Interaction Patterns in Lane-Change Scenarios
• Modeling Continuous Stochastic Processes with Dynamic Normalizing Flows
• Few-shot Learning for Time-series Forecasting
• Context-Aware Safe Reinforcement Learning for Non-Stationary Environments
• Few-Shot Learning Patterns in Financial Time-Series for Trend-Following Strategies
• When in Doubt: Neural Non-Parametric Uncertainty Quantification for Epidemic Forecasting
• Task-Agnostic Online Reinforcement Learning with an Infinite Mixture of Gaussian Processes
• Meta-Learning for Koopman Spectral Analysis with Short Time-series
• Robustifying Sequential Neural Processes