Improved Prosody from Learned F0 Codebook Representations for VQ-VAE Speech Waveform Reconstruction
arXiv:2005.07884
Abstract
Vector Quantized Variational AutoEncoders (VQ-VAE) are a powerful representation learning framework that can discover discrete groups of features from a speech signal without supervision. Until now, the VQ-VAE architecture has previously modeled individual types of speech features, such as only phones or only F0. This paper introduces an important extension to VQ-VAE for learning F0-related suprasegmental information simultaneously along with traditional phone features.The proposed framework uses two encoders such that the F0 trajectory and speech waveform are both input to the system, therefore two separate codebooks are learned. We used a WaveRNN vocoder as the decoder component of VQ-VAE. Our speaker-independent VQ-VAE was trained with raw speech waveforms from multi-speaker Japanese speech databases. Experimental results show that the proposed extension reduces F0 distortion of reconstructed speech for all unseen test speakers, and results in significantly higher preference scores from a listening test. We additionally conducted experiments using single-speaker Mandarin speech to demonstrate advantages of our architecture in another language which relies heavily on F0.
Submitted to Interspeech 2020
References in corpus (9)
- Adam: A Method for Stochastic Optimization
- Neural Discrete Representation Learning
- Low Bit-Rate Speech Coding with VQ-VAE and a WaveNet Decoder
- Jukebox: A Generative Model for Music
- Efficient Neural Audio Synthesis
- Deep Encoder-Decoder Models for Unsupervised Learning of Controllable Speech Synthesis
- Unsupervised Speech Decomposition via Triple Information Bottleneck
- JVS corpus: free Japanese multi-speaker voice corpus
- VQVAE Unsupervised Unit Discovery and Multi-scale Code2Spec Inverter for Zerospeech Challenge 2019