A Comparative Study of Self-supervised Speech Representation Based Voice Conversion
arXiv:2207.04356 · doi:10.1109/JSTSP.2022.3193761
Abstract
We present a large-scale comparative study of self-supervised speech representation (S3R)-based voice conversion (VC). In the context of recognition-synthesis VC, S3Rs are attractive owing to their potential to replace expensive supervised representations such as phonetic posteriorgrams (PPGs), which are commonly adopted by state-of-the-art VC systems. Using S3PRL-VC, an open-source VC software we previously developed, we provide a series of in-depth objective and subjective analyses under three VC settings: intra-/cross-lingual any-to-one (A2O) and any-to-any (A2A) VC, using the voice conversion challenge 2020 (VCC2020) dataset. We investigated S3R-based VC in various aspects, including model type, multilinguality, and supervision. We also studied the effect of a post-discretization process with k-means clustering and showed how it improves in the A2A setting. Finally, the comparison with state-of-the-art VC systems demonstrates the competitiveness of S3R-based VC and also sheds light on the possible improving directions.
arXiv commentAccepted to IEEE Journal of Selected Topics in Signal Processing. arXiv admin note: substantial text overlap with arXiv:2110.06280
References in corpus (11)
- Auto-Encoding Variational Bayes
- Representation Learning with Contrastive Predictive Coding
- VoxCeleb2: Deep Speaker Recognition
- Libri-Light: A Benchmark for ASR with Limited or No Supervision
- Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoders
- TERA: Self-Supervised Learning of Transformer Encoder Representation for Speech
- Deep Contextualized Acoustic Representations For Semi-Supervised Speech Recognition
- Sequence-to-Sequence Acoustic Modeling for Voice Conversion
- An Open source Implementation of ITU-T Recommendation P.808 with Validation
- Non-Parallel Sequence-to-Sequence Voice Conversion with Disentangled Linguistic and Speaker Representations
- Refined WaveNet Vocoder for Variational Autoencoder Based Voice Conversion