Unsupervised End-to-End Learning of Discrete Linguistic Units for Voice Conversion

Abstract

We present an unsupervised end-to-end training scheme where we discover discrete subword units from speech without using any labels. The discrete subword units are learned under an ASR-TTS autoencoder reconstruction setting, where an ASR-Encoder is trained to discover a set of common linguistic units given a variety of speakers, and a TTS-Decoder trained to project the discovered units back to the designated speech. We propose a discrete encoding method, Multilabel-Binary Vectors (MBV), to make the ASR-TTS autoencoder differentiable. We found that the proposed encoding method offers automatic extraction of speech content from speaker style, and is sufficient to cover full linguistic content in a given language. Therefore, the TTS-Decoder can synthesize speech with the same content as the input of ASR-Encoder but with different speaker characteristics, which achieves voice conversion (VC). We further improve the quality of VC using adversarial training, where we train a TTS-Patcher that augments the output of TTS-Decoder. Objective and subjective evaluations show that the proposed approach offers strong VC results as it eliminates speaker identity while preserving content within speech. In the ZeroSpeech 2019 Challenge, we achieved outstanding performance in terms of low bitrate.

arXiv commentAccepted by Interspeech 2019, Graz, Austria

References in corpus (10)

• Adam: A Method for Stochastic Optimization
• Categorical Reparameterization with Gumbel-Softmax
• Neural Discrete Representation Learning
• Conditional Image Synthesis With Auxiliary Classifier GANs
• Wasserstein GAN
• Unsupervised Learning of Disentangled and Interpretable Representations from Sequential Data
• Tacotron: Towards End-to-End Speech Synthesis
• Deep Voice 3: Scaling Text-to-Speech with Convolutional Sequence Learning
• Discriminative Acoustic Word Embeddings: Recurrent Neural Network-Based Approaches
• Improved Audio Embeddings by Adjacency-Based Clustering with Applications in Spoken Term Detection

Cited by in corpus (12)

• Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoders
• TERA: Self-Supervised Learning of Transformer Encoder Representation for Speech
• Self-supervised language learning from raw audio: Lessons from the Zero Resource Speech Challenge
• VQVC+: One-Shot Voice Conversion by Vector Quantization and U-Net architecture
• Decentralized policy learning with partial observation and mechanical constraints for multiperson modeling
• Learning Hierarchical Discrete Linguistic Units from Visually-Grounded Speech
• Bootstrapping non-parallel voice conversion from speaker-adaptive text-to-speech
• Parallel Synthesis for Autoregressive Speech Generation
• Adversarially learning disentangled speech representations for robust multi-factor voice conversion
• Speech-to-speech Translation between Untranscribed Unknown Languages
• Exploration of End-to-end Synthesisers forZero Resource Speech Challenge 2020
• How Far Are We from Robust Voice Conversion: A Survey