A Comparison of Discrete and Soft Speech Units for Improved Voice Conversion

Abstract

The goal of voice conversion is to transform source speech into a target voice, keeping the content unchanged. In this paper, we focus on self-supervised representation learning for voice conversion. Specifically, we compare discrete and soft speech units as input features. We find that discrete representations effectively remove speaker information but discard some linguistic content - leading to mispronunciations. As a solution, we propose soft speech units. To learn soft units, we predict a distribution over discrete speech units. By modeling uncertainty, soft units capture more content information, improving the intelligibility and naturalness of converted speech. Samples available at https://ubisoft-laforge.github.io/speech/soft-vc/. Code available at https://github.com/bshall/soft-vc/.

5 pages, 2 figures, 2 tables. Accepted at ICASSP 2022

References in corpus (6)

• WaveNet: A Generative Model for Raw Audio
• HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis
• AUTOVC: Zero-Shot Voice Style Transfer with Only Autoencoder Loss
• The Zero Resource Speech Benchmark 2021: Metrics and baselines for unsupervised spoken language modeling
• Voice Conversion Challenge 2020: Intra-lingual semi-parallel and cross-lingual voice conversion
• The Sequence-to-Sequence Baseline for the Voice Conversion Challenge 2020: Cascading ASR and TTS

Cited by in corpus (3)

• WESPER: Zero-shot and Realtime Whisper to Normal Voice Conversion for Whisper-based Speech Interactions
• DurFlex-EVC: Duration-Flexible Emotional Voice Conversion Leveraging Discrete Representations without Text Alignment
• Quantifying Source Speaker Leakage in One-to-One Voice Conversion