Learning Stochastic Recurrent Networks

Abstract

Leveraging advances in variational inference, we propose to enhance recurrent neural networks with latent variables, resulting in Stochastic Recurrent Networks (STORNs). The model i) can be trained with stochastic gradient methods, ii) allows structured and multi-modal conditionals at each time step, iii) features a reliable estimator of the marginal likelihood and iv) is a generalisation of deterministic recurrent neural networks. We evaluate the method on four polyphonic musical data sets and motion capture data.

Submitted to conference track of ICLR 2015

References in corpus (7)

• ADADELTA: An Adaptive Learning Rate Method
• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• Modeling Temporal Dependencies in High-Dimensional Sequences: Application to Polyphonic Music Generation and Transcription
• How to Construct Deep Recurrent Neural Networks
• On Fast Dropout and its Applicability to Recurrent Networks
• Speech Recognition with Deep Recurrent Neural Networks

Cited by in corpus (70)

• An Introduction to Variational Autoencoders
• Recurrent Neural Networks for Time Series Forecasting: Current Status and Future Directions
• A Recurrent Latent Variable Model for Sequential Data
• TensorFlow Distributions
• Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Images
• Dynamical Variational Autoencoders: A Comprehensive Review
• Variational Recurrent Auto-Encoders
• Bayesian Recurrent Neural Networks
• Deep Kalman Filters
• Deep learning algorithm for data-driven simulation of noisy dynamical system
• Hierarchical Implicit Models and Likelihood-Free Variational Inference
• Improved Variational Autoencoders for Text Modeling using Dilated Convolutions
• Modeling emotion in complex stories: the Stanford Emotional Narratives Dataset
• Identifying nonlinear dynamical systems via generative recurrent neural networks with applications to fMRI
• Deep Variational Bayes Filters: Unsupervised Learning of State Space Models from Raw Data
• CLVSA: A Convolutional LSTM Based Variational Sequence-to-Sequence Model with Attention for Predicting Trends of Financial Markets
• Spatio-Temporal Neural Networks for Space-Time Series Forecasting and Relations Discovery
• Physics-guided Deep Markov Models for Learning Nonlinear Dynamical Systems with Uncertainty
• Breaking the Softmax Bottleneck: A High-Rank RNN Language Model
• Deep Temporal Sigmoid Belief Networks for Sequence Modeling
• Neural Adaptive Sequential Monte Carlo
• Generative Temporal Models with Memory
• Z-Forcing: Training Stochastic Recurrent Networks
• State Space LSTM Models with Particle MCMC Inference
• Toward Diverse Text Generation with Inverse Reinforcement Learning
• Noisin: Unbiased Regularization for Recurrent Neural Networks
• Stochastic WaveNet: A Generative Latent Variable Model for Sequential Data
• Perpetual Motion: Generating Unbounded Human Motion
• Bidirectional Recurrent Neural Networks as Generative Models - Reconstructing Gaps in Time Series
• Learning Dynamics Model in Reinforcement Learning by Incorporating the Long Term Future
• Neural Net Models for Open-Domain Discourse Coherence
• Disentangled State Space Representations
• Continual Learning in Recurrent Neural Networks
• Recurrent Neural Processes
• Temporal Difference Variational Auto-Encoder
• Social-VRNN: One-Shot Multi-modal Trajectory Prediction for Interacting Pedestrians
• Forecasting Individualized Disease Trajectories using Interpretable Deep Learning
• A Classifying Variational Autoencoder with Application to Polyphonic Music Generation
• Learning Awareness Models
• State-Regularized Recurrent Neural Networks
• Multi-space Variational Encoder-Decoders for Semi-supervised Labeled Sequence Transduction
• Recurrent Neural Networks with Stochastic Layers for Acoustic Novelty Detection
• Innovations Autoencoder and its Application in One-class Anomalous Sequence Detection
• S3VAE: Self-Supervised Sequential VAE for Representation Disentanglement and Data Generation
• Variational Bi-LSTMs
• A Latent Feelings-aware RNN Model for User Churn Prediction with Behavioral Data
• Anomaly Detection on Graph Time Series
• Factored Temporal Sigmoid Belief Networks for Sequence Learning
• Improving Sequential Latent Variable Models with Autoregressive Flows
• Deep Probabilistic Time Series Forecasting using Augmented Recurrent Input for Dynamic Systems
• Stochastic Sequential Neural Networks with Structured Inference
• Variational Tracking and Prediction with Generative Disentangled State-Space Models
• Bidirectional Inference Networks: A Class of Deep Bayesian Networks for Health Profiling
• Human Pose Forecasting via Deep Markov Models
• Unsupervised Contextual Anomaly Detection using Joint Deep Variational Generative Models
• Relational State-Space Model for Stochastic Multi-Object Systems
• Regularized Sequential Latent Variable Models with Adversarial Neural Networks
• Policy Gradients Incorporating the Future
• Mind the Gap when Conditioning Amortised Inference in Sequential Latent-Variable Models
• Re-examination of the Role of Latent Variables in Sequence Modeling
• Ensemble Kalman Variational Objectives: Nonlinear Latent Trajectory Inference with A Hybrid of Variational Inference and Ensemble Kalman Filter
• Bayesian Attention Modules
• Beta DVBF: Learning State-Space Models for Control from High Dimensional Observations
• Hierarchical Variational Imitation Learning of Control Programs
• Switching Recurrent Kalman Networks
• Sequential Variational Autoencoders for Collaborative Filtering
• A Benchmark of Dynamical Variational Autoencoders applied to Speech Spectrogram Modeling
• Dual-CLVSA: a Novel Deep Learning Approach to Predict Financial Markets with Sentiment Measurements
• Anomaly Detection of Time Series with Smoothness-Inducing Sequential Variational Auto-Encoder
• Introducing a Generative Adversarial Network Model for Lagrangian Trajectory Simulation