A Recurrent Latent Variable Model for Sequential Data

Abstract

In this paper, we explore the inclusion of latent random variables into the dynamic hidden state of a recurrent neural network (RNN) by combining elements of the variational autoencoder. We argue that through the use of high-level latent random variables, the variational RNN (VRNN)1 can model the kind of variability observed in highly structured sequential data such as natural speech. We empirically evaluate the proposed model against related sequential models on four speech datasets and one handwriting dataset. Our results show the important roles that latent random variables can play in the RNN dynamic hidden state.

References in corpus (4)

• Theano: new features and speed improvements
• Modeling Temporal Dependencies in High-Dimensional Sequences: Application to Polyphonic Music Generation and Transcription
• Learning Stochastic Recurrent Networks
• Variational Recurrent Auto-Encoders

Cited by in corpus (198)

• An Introduction to Variational Autoencoders
• Adaptive Graph Convolutional Recurrent Network for Traffic Forecasting
• Human-level performance in first-person multiplayer games with population-based deep reinforcement learning
• An Algorithmic Perspective on Imitation Learning
• A Review on Deep Learning Techniques for Video Prediction
• A Hierarchical Latent Vector Model for Learning Long-Term Structure in Music
• Deep Learning for Physical Processes: Incorporating Prior Scientific Knowledge
• Dynamical Variational Autoencoders: A Comprehensive Review
• Bayesian Recurrent Neural Networks
• Statistical Speech Enhancement Based on Probabilistic Integration of Variational Autoencoder and Non-Negative Matrix Factorization
• GeoTrackNet-A Maritime Anomaly Detector using Probabilistic Neural Network Representation of AIS Tracks and A Contrario Detection
• A Multi-task Deep Learning Architecture for Maritime Surveillance using AIS Data Streams
• Sequential Latent Knowledge Selection for Knowledge-Grounded Dialogue
• Deep learning algorithm for data-driven simulation of noisy dynamical system
• Lifelong Generative Modeling
• Time Series Anomaly Detection for Cyber-Physical Systems via Neural System Identification and Bayesian Filtering
• Few-Shot Deep Adversarial Learning for Video-based Person Re-identification
• ODE${}^2$VAE: Deep generative second order ODEs with Bayesian neural networks
• Modeling emotion in complex stories: the Stanford Emotional Narratives Dataset
• Learning Plannable Representations with Causal InfoGAN
• Collaborative Recurrent Autoencoder: Recommend while Learning to Fill in the Blanks
• Identifying nonlinear dynamical systems via generative recurrent neural networks with applications to fMRI
• An Improved Multi-Output Gaussian Process RNN with Real-Time Validation for Early Sepsis Detection
• Recurrent Independent Mechanisms
• Neural Abstractive Text Summarization with Sequence-to-Sequence Models
• Spatio-Temporal Neural Networks for Space-Time Series Forecasting and Relations Discovery
• Variational Temporal Deep Generative Model for Radar HRRP Target Recognition
• Physics-guided Deep Markov Models for Learning Nonlinear Dynamical Systems with Uncertainty
• Deep Temporal Sigmoid Belief Networks for Sequence Modeling
• ClariNet: Parallel Wave Generation in End-to-End Text-to-Speech
• Deep Factors for Forecasting
• Noisy Parallel Approximate Decoding for Conditional Recurrent Language Model
• Variational Transformers for Diverse Response Generation
• VectorNet: Encoding HD Maps and Agent Dynamics from Vectorized Representation
• AC-VRNN: Attentive Conditional-VRNN for Multi-Future Trajectory Prediction
• Goal-Directed Planning for Habituated Agents by Active Inference Using a Variational Recurrent Neural Network
• Disentangled Variational Autoencoder based Multi-Label Classification with Covariance-Aware Multivariate Probit Model
• FitVid: Overfitting in Pixel-Level Video Prediction
• Structured Attention for Unsupervised Dialogue Structure Induction
• Toward a Better Monitoring Statistic for Profile Monitoring via Variational Autoencoders
• Learning to Set Waypoints for Audio-Visual Navigation
• Probabilistic Character Motion Synthesis using a Hierarchical Deep Latent Variable Model
• EM-like Learning Chaotic Dynamics from Noisy and Partial Observations
• Discriminative Particle Filter Reinforcement Learning for Complex Partial Observations
• Recurrent Attentive Neural Process for Sequential Data
• Neural Language Generation: Formulation, Methods, and Evaluation
• Augmented Normalizing Flows: Bridging the Gap Between Generative Flows and Latent Variable Models
• Autoencoders for music sound modeling: a comparison of linear, shallow, deep, recurrent and variational models
• PIANOTREE VAE: Structured Representation Learning for Polyphonic Music
• Variational Recurrent Models for Solving Partially Observable Control Tasks
• Deep Generative Video Compression
• Structured Object-Aware Physics Prediction for Video Modeling and Planning
• Modeling Continuous Stochastic Processes with Dynamic Normalizing Flows
• Learning Dynamics Model in Reinforcement Learning by Incorporating the Long Term Future
• Generative Adversarial Network for Handwritten Text
• Unsupervised Learning of Object Structure and Dynamics from Videos
• A Survey on Bayesian Deep Learning
• Graph Generation with Variational Recurrent Neural Network
• Disentangled Recurrent Wasserstein Autoencoder
• From Deterministic to Generative: Multi-Modal Stochastic RNNs for Video Captioning
• Multi-period Time Series Modeling with Sparsity via Bayesian Variational Inference
• Gaussian Mixture Variational Autoencoder with Contrastive Learning for Multi-Label Classification
• Neural Dynamics Discovery via Gaussian Process Recurrent Neural Networks
• Spatio-Temporal Handwriting Imitation
• VARA-TTS: Non-Autoregressive Text-to-Speech Synthesis based on Very Deep VAE with Residual Attention
• MILD: Multimodal Interactive Latent Dynamics for Learning Human-Robot Interaction
• Contrastively Disentangled Sequential Variational Autoencoder
• Behavior Priors for Efficient Reinforcement Learning
• Disentangled State Space Representations
• Preventing Posterior Collapse with delta-VAEs
• Continual Learning in Recurrent Neural Networks
• Decentralized policy learning with partial observation and mechanical constraints for multiperson modeling
• Unsupervised Dialog Structure Learning
• Adversarial Domain Adaptation for Variational Neural Language Generation in Dialogue Systems
• Variational Laplace Autoencoders
• Clockwork Variational Autoencoders
• Social-VRNN: One-Shot Multi-modal Trajectory Prediction for Interacting Pedestrians
• Ball Trajectory Inference from Multi-Agent Sports Contexts Using Set Transformer and Hierarchical Bi-LSTM
• Temporal Difference Variational Auto-Encoder
• Estimating the Euclidean quantum propagator with deep generative modeling of Feynman paths
• Future Frame Prediction Using Convolutional VRNN for Anomaly Detection
• Differentiable Particle Filtering via Entropy-Regularized Optimal Transport
• Latent Representation in Human-Robot Interaction with Explicit Consideration of Periodic Dynamics
• A Classifying Variational Autoencoder with Application to Polyphonic Music Generation
• What went wrong and when? Instance-wise Feature Importance for Time-series Models
• Recurrent Neural Networks with Stochastic Layers for Acoustic Novelty Detection
• Information Maximizing Visual Question Generation
• Physics-Integrated Variational Autoencoders for Robust and Interpretable Generative Modeling
• MetaCURE: Meta Reinforcement Learning with Empowerment-Driven Exploration
• Generating Realistic Stock Market Order Streams
• baller2vec: A Multi-Entity Transformer For Multi-Agent Spatiotemporal Modeling
• S3VAE: Self-Supervised Sequential VAE for Representation Disentanglement and Data Generation
• An Adversarial Domain Separation Framework for Septic Shock Early Prediction Across EHR Systems
• Unsupervised Video Decomposition using Spatio-temporal Iterative Inference
• Exploring Spatial-Temporal Multi-Frequency Analysis for High-Fidelity and Temporal-Consistency Video Prediction
• Learning Interpretable Deep State Space Model for Probabilistic Time Series Forecasting
• MissFormer: (In-)attention-based handling of missing observations for trajectory filtering and prediction
• SoundSpaces: Audio-Visual Navigation in 3D Environments
• Contrastive Variational Reinforcement Learning for Complex Observations
• Learning representations for multivariate time series with missing data using Temporal Kernelized Autoencoders
• H-VGRAE: A Hierarchical Stochastic Spatial-Temporal Embedding Method for Robust Anomaly Detection in Dynamic Networks
• Learning Disentangled Representations for Time Series
• Hierarchical Autoregressive Modeling for Neural Video Compression
• Variational Deep Learning for the Identification and Reconstruction of Chaotic and Stochastic Dynamical Systems from Noisy and Partial Observations
• Simple Video Generation using Neural ODEs
• Improving Sequential Latent Variable Models with Autoregressive Flows
• PLSO: A generative framework for decomposing nonstationary time-series into piecewise stationary oscillatory components
• Benchmarking Deep Sequential Models on Volatility Predictions for Financial Time Series
• A Brief Overview of Unsupervised Neural Speech Representation Learning
• Vid2Param: Modelling of Dynamics Parameters from Video
• Probabilistic Video Generation using Holistic Attribute Control
• A survey on Variational Autoencoders from a GreenAI perspective
• Few Shot System Identification for Reinforcement Learning
• HireVAE: An Online and Adaptive Factor Model Based on Hierarchical and Regime-Switch VAE
• Dynamic Future Net: Diversified Human Motion Generation
• On Predictive Information in RNNs
• Bayesian Learning of LF-MMI Trained Time Delay Neural Networks for Speech Recognition
• Optimization Algorithm for Feedback and Feedforward Policies towards Robot Control Robust to Sensing Failures
• Agent Modelling under Partial Observability for Deep Reinforcement Learning
• Self-organization of action hierarchy and compositionality by reinforcement learning with recurrent neural networks
• The Monte Carlo Transformer: a stochastic self-attention model for sequence prediction
• Vertical-Horizontal Structured Attention for Generating Music with Chords
• Goal-Directed Planning by Reinforcement Learning and Active Inference
• Pixyz: a Python library for developing deep generative models
• Learning Nonlinear State Space Models with Hamiltonian Sequential Monte Carlo Sampler
• Variational Hyper RNN for Sequence Modeling
• A Stochastic Decoder for Neural Machine Translation
• Transferable Time-Series Forecasting under Causal Conditional Shift
• A Variational Time Series Feature Extractor for Action Prediction
• Variational Tracking and Prediction with Generative Disentangled State-Space Models
• Style Equalization: Unsupervised Learning of Controllable Generative Sequence Models
• Stochastic Sequential Neural Networks with Structured Inference
• Modeling Semantic Relationship in Multi-turn Conversations with Hierarchical Latent Variables
• Deep learning reveals hidden interactions in complex systems
• Energy-Inspired Models: Learning with Sampler-Induced Distributions
• Information-Theoretic Odometry Learning
• Monte Carlo Filtering Objectives: A New Family of Variational Objectives to Learn Generative Model and Neural Adaptive Proposal for Time Series
• Variational Dynamic Mixtures
• Time-series Imputation of Temporally-occluded Multiagent Trajectories
• Learning Disentangled Representations of Video with Missing Data
• Guiding Variational Response Generator to Exploit Persona
• Sequential Adversarial Anomaly Detection for One-Class Event Data
• Semi-supervised Sequential Generative Models
• OCEAN: Online Task Inference for Compositional Tasks with Context Adaptation
• Improving Fair Predictions Using Variational Inference In Causal Models
• Variational Marginal Particle Filters
• Relational State-Space Model for Stochastic Multi-Object Systems
• Mind the Gap when Conditioning Amortised Inference in Sequential Latent-Variable Models
• Ensemble Kalman Variational Objectives: Nonlinear Latent Trajectory Inference with A Hybrid of Variational Inference and Ensemble Kalman Filter
• Semi-Supervised Dialogue Policy Learning via Stochastic Reward Estimation
• Optimizing a quantum reservoir computer for time series prediction
• Variational Sequential Labelers for Semi-Supervised Learning
• Bayesian neural networks and dimensionality reduction
• Imitation Learning of Factored Multi-agent Reactive Models
• Beta DVBF: Learning State-Space Models for Control from High Dimensional Observations
• Working Memory Graphs
• Bayesian Attention Modules
• Hierarchical Variational Imitation Learning of Control Programs
• Deep Switching State Space Model (DS${}^3$M) for Nonlinear Time Series Forecasting with Regime Switching
• Improving Lossless Compression Rates via Monte Carlo Bits-Back Coding
• Detection of Lying Electrical Vehicles in Charging Coordination Application Using Deep Learning
• Memory and attention in deep learning
• Re-examination of the Role of Latent Variables in Sequence Modeling
• Hierarchical Video Generation for Complex Data
• Dynamic Relational Inference in Multi-Agent Trajectories
• Regularized Sequential Latent Variable Models with Adversarial Neural Networks
• Latent-Variable Generative Models for Data-Efficient Text Classification
• Representation Learning for Sequence Data with Deep Autoencoding Predictive Components
• Online Variational Filtering and Parameter Learning
• From abstract items to latent spaces to observed data and back: Compositional Variational Auto-Encoder
• Variational State-Space Models for Localisation and Dense 3D Mapping in 6 DoF
• Action2video: Generating Videos of Human 3D Actions
• Analysis of ODE2VAE with Examples
• Recurrent Flow Networks: A Recurrent Latent Variable Model for Density Modelling of Urban Mobility
• Deep Variational Luenberger-type Observer for Stochastic Video Prediction
• DSBERT:Unsupervised Dialogue Structure learning with BERT
• A Temporal Variational Model for Story Generation
• Probability Trajectory: One New Movement Description for Trajectory Prediction
• Discrete Auto-regressive Variational Attention Models for Text Modeling
• Anomaly Detection of Time Series with Smoothness-Inducing Sequential Variational Auto-Encoder
• Deep Stochastic Volatility Model
• A Benchmark of Dynamical Variational Autoencoders applied to Speech Spectrogram Modeling
• Constructing Gradient Controllable Recurrent Neural Networks Using Hamiltonian Dynamics
• Uncertainty-Gated Stochastic Sequential Model for EHR Mortality Prediction
• Medical data wrangling with sequential variational autoencoders
• Semi-Implicit Stochastic Recurrent Neural Networks
• An RNN-based IMM Filter Surrogate
• Detection of Abnormal Vessel Behaviours from AIS data using GeoTrackNet: from the Laboratory to the Ocean
• A Variational Auto-Encoder Model for Stochastic Point Processes
• Switching Recurrent Kalman Networks
• Rényi Divergence in General Hidden Markov Models
• Continuous Latent Process Flows
• Generating Handwriting via Decoupled Style Descriptors
• 3D Conceptual Design Using Deep Learning
• Causal Mechanism Transfer Network for Time Series Domain Adaptation in Mechanical Systems
• SmartPatch: Improving Handwritten Word Imitation with Patch Discriminators
• Diversifying Topic-Coherent Response Generation for Natural Multi-turn Conversations
• SchrödingeRNN: Generative Modeling of Raw Audio as a Continuously Observed Quantum State