Stochastic WaveNet: A Generative Latent Variable Model for Sequential Data

Abstract

How to model distribution of sequential data, including but not limited to speech and human motions, is an important ongoing research problem. It has been demonstrated that model capacity can be significantly enhanced by introducing stochastic latent variables in the hidden states of recurrent neural networks. Simultaneously, WaveNet, equipped with dilated convolutions, achieves astonishing empirical performance in natural speech generation task. In this paper, we combine the ideas from both stochastic latent variables and dilated convolutions, and propose a new architecture to model sequential data, termed as Stochastic WaveNet, where stochastic latent variables are injected into the WaveNet structure. We argue that Stochastic WaveNet enjoys powerful distribution modeling capacity and the advantage of parallel training from dilated convolutions. In order to efficiently infer the posterior distribution of the latent variables, a novel inference network structure is designed based on the characteristics of WaveNet architecture. State-of-the-art performances on benchmark datasets are obtained by Stochastic WaveNet on natural speech modeling and high quality human handwriting samples can be generated as well.

arXiv commentICML 2018 Workshop

References in corpus (16)

• Adam: A Method for Stochastic Optimization
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• WaveNet: A Generative Model for Raw Audio
• Pixel Recurrent Neural Networks
• Modeling Temporal Dependencies in High-Dimensional Sequences: Application to Polyphonic Music Generation and Transcription
• Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders
• Learning Stochastic Recurrent Networks
• Variational Recurrent Auto-Encoders
• Sequential Neural Models with Stochastic Layers
• Improved Variational Autoencoders for Text Modeling using Dilated Convolutions
• Breaking the Softmax Bottleneck: A High-Rank RNN Language Model
• Deep Temporal Sigmoid Belief Networks for Sequence Modeling
• A Hybrid Convolutional Variational Autoencoder for Text Generation
• Neural Adaptive Sequential Monte Carlo
• Z-Forcing: Training Stochastic Recurrent Networks
• Variational Bi-LSTMs

Cited by in corpus (4)

• Cyclical Annealing Schedule: A Simple Approach to Mitigating KL Vanishing
• Re-examination of the Role of Latent Variables in Sequence Modeling
• Semi-Implicit Stochastic Recurrent Neural Networks
• A Fully Time-domain Neural Model for Subband-based Speech Synthesizer