The Monte Carlo Transformer: a stochastic self-attention model for sequence prediction
arXiv:2007.08620
Abstract
This paper introduces the Sequential Monte Carlo Transformer, an original approach that naturally captures the observations distribution in a transformer architecture. The keys, queries, values and attention vectors of the network are considered as the unobserved stochastic states of its hidden structure. This generative model is such that at each time step the received observation is a random function of its past states in a given attention window. In this general state-space setting, we use Sequential Monte Carlo methods to approximate the posterior distributions of the states given the observations, and to estimate the gradient of the log-likelihood. We hence propose a generative model giving a predictive distribution, instead of a single-point estimate.
References in corpus (21)
- Auto-Encoding Variational Bayes
- Neural Machine Translation by Jointly Learning to Align and Translate
- Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning
- A Structured Self-attentive Sentence Embedding
- Weight Uncertainty in Neural Networks
- Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting
- A Recurrent Latent Variable Model for Sequential Data
- Noisy Networks for Exploration
- Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks
- Deep Exploration via Bootstrapped DQN
- Deep and Confident Prediction for Time Series at Uber
- Bayesian Uncertainty Estimation for Batch Normalized Deep Networks
- Sequential Monte Carlo smoothing with application to parameter estimation in non-linear state space models
- Variational Recurrent Auto-Encoders
- Sequential Neural Models with Stochastic Layers
- High-Quality Prediction Intervals for Deep Learning: A Distribution-Free, Ensembled Approach
- Bayesian Recurrent Neural Networks
- Kalman Temporal Differences
- Variational Sequential Monte Carlo
- On the Expressiveness of Approximate Inference in Bayesian Neural Networks
- Variational Hyper RNN for Sequence Modeling