Dynamical Variational Autoencoders: A Comprehensive Review

Abstract

Variational autoencoders (VAEs) are powerful deep generative models widely used to represent high-dimensional complex data through a low-dimensional latent space learned in an unsupervised manner. In the original VAE model, the input data vectors are processed independently. Recently, a series of papers have presented different extensions of the VAE to process sequential data, which model not only the latent space but also the temporal dependencies within a sequence of data vectors and corresponding latent vectors, relying on recurrent neural networks or state-space models. In this paper, we perform a literature review of these models. We introduce and discuss a general class of models, called dynamical variational autoencoders (DVAEs), which encompasses a large subset of these temporal VAE extensions. Then, we present in detail seven recently proposed DVAE models, with an aim to homogenize the notations and presentation lines, as well as to relate these models with existing classical temporal models. We have reimplemented those seven DVAE models and present the results of an experimental benchmark conducted on the speech analysis-resynthesis task (the PyTorch code is made publicly available). The paper concludes with a discussion on important issues concerning the DVAE class of models and future research guidelines.

References in corpus (52)

• Adam: A Method for Stochastic Optimization
• Auto-Encoding Variational Bayes
• TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
• An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling
• WaveNet: A Generative Model for Raw Audio
• Neural Discrete Representation Learning
• An Introduction to Variational Autoencoders
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separation
• Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
• Variational Inference with Normalizing Flows
• NIPS 2016 Tutorial: Generative Adversarial Networks
• A Recurrent Latent Variable Model for Sequential Data
• Disentangling by Factorising
• Modeling Temporal Dependencies in High-Dimensional Sequences: Application to Polyphonic Music Generation and Transcription
• Decomposing Motion and Content for Natural Video Sequence Prediction
• NVAE: A Deep Hierarchical Variational Autoencoder
• Markov Chain Monte Carlo and Variational Inference: Bridging the Gap
• Stochastic Video Generation with a Learned Prior
• A Hierarchical Latent Vector Model for Learning Long-Term Structure in Music
• Auxiliary Deep Generative Models
• Bayesian Nonparametric Inference of Switching Linear Dynamical Systems
• Video Compression With Rate-Distortion Autoencoders
• Unsupervised Learning of Disentangled Representations from Video
• Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Images
• Composing graphical models with neural networks for structured representations and fast inference
• Neural Variational Inference for Text Processing
• Learning Stochastic Recurrent Networks
• A Flexible and Efficient Algorithmic Framework for Constrained Matrix and Tensor Factorization
• Variational Recurrent Auto-Encoders
• Sequential Neural Models with Stochastic Layers
• Unsupervised Learning of Disentangled and Interpretable Representations from Sequential Data
• Learning Disentangled Representations with Semi-Supervised Deep Generative Models
• Statistical Speech Enhancement Based on Probabilistic Integration of Variational Autoencoder and Non-Negative Matrix Factorization
• Lagging Inference Networks and Posterior Collapse in Variational Autoencoders
• Low Bit-Rate Speech Coding with VQ-VAE and a WaveNet Decoder
• A Disentangled Recognition and Nonlinear Dynamics Model for Unsupervised Learning
• Black box variational inference for state space models
• Towards Conceptual Compression
• Disentangled Sequential Autoencoder
• Semi-supervised multichannel speech enhancement with variational autoencoders and non-negative matrix factorization
• Deep Temporal Sigmoid Belief Networks for Sequence Modeling
• Variational Sequential Monte Carlo
• Tensor Factorization via Matrix Factorization
• Recurrent switching linear dynamical systems
• Tackling Over-pruning in Variational Autoencoders
• Semi-blind source separation with multichannel variational autoencoder
• The Usual Suspects? Reassessing Blame for VAE Posterior Collapse
• A Sober Look at the Unsupervised Learning of Disentangled Representations and their Evaluation
• Deep Generative Video Compression
• Neural Granular Sound Synthesis
• Variational online learning of neural dynamics

Cited by in corpus (11)

• Self-Supervised Speech Representation Learning: A Review
• A Survey of Sound Source Localization with Deep Learning Methods
• Speech Enhancement and Dereverberation with Diffusion-based Generative Models
• DANSE: Data-driven Non-linear State Estimation of Model-free Process in Unsupervised Learning Setup
• A multimodal dynamical variational autoencoder for audiovisual speech representation learning
• Searching for Anomalies in the ZTF Catalog of Periodic Variable Stars
• Learning and controlling the source-filter representation of speech with a variational autoencoder
• Deep learning and differential equations for modeling changes in individual-level latent dynamics between observation periods
• From reductionism to realism: Holistic mathematical modelling for complex biological systems
• Variational Autoencoder for Calibration: A New Approach
• Using matrix-product states for time-series machine learning