Re-examination of the Role of Latent Variables in Sequence Modeling
arXiv:1902.01388
Abstract
With latent variables, stochastic recurrent models have achieved state-of-the-art performance in modeling sound-wave sequence. However, opposite results are also observed in other domains, where standard recurrent networks often outperform stochastic models. To better understand this discrepancy, we re-examine the roles of latent variables in stochastic recurrent models for speech density estimation. Our analysis reveals that under the restriction of fully factorized output distribution in previous evaluations, the stochastic models were implicitly leveraging intra-step correlation but the standard recurrent baselines were prohibited to do so, resulting in an unfair comparison. To correct the unfairness, we remove such restriction in our re-examination, where all the models can explicitly leverage intra-step correlation with an auto-regressive structure. Over a diverse set of sequential data, including human speech, MIDI music, handwriting trajectory and frame-permuted speech, our results show that stochastic recurrent models fail to exhibit any practical advantage despite the claimed theoretical superiority. In contrast, standard recurrent models equipped with an auto-regressive output distribution consistently perform better, significantly advancing the state-of-the-art results on three speech datasets.
arXiv commentCode available at https://github.com/zihangdai/reexamine-srnn, accepted by NeurIPS 2019
References in corpus (15)
- Auto-Encoding Variational Bayes
- Stochastic Backpropagation and Approximate Inference in Deep Generative Models
- Conditional Image Generation with PixelCNN Decoders
- SGDR: Stochastic Gradient Descent with Warm Restarts
- Pixel Recurrent Neural Networks
- Variational Inference with Normalizing Flows
- DRAW: A Recurrent Neural Network For Image Generation
- Density estimation using Real NVP
- A Recurrent Latent Variable Model for Sequential Data
- Modeling Temporal Dependencies in High-Dimensional Sequences: Application to Polyphonic Music Generation and Transcription
- MADE: Masked Autoencoder for Distribution Estimation
- Learning Stochastic Recurrent Networks
- Deep Kalman Filters
- A Disentangled Recognition and Nonlinear Dynamics Model for Unsupervised Learning
- Stochastic WaveNet: A Generative Latent Variable Model for Sequential Data