Any-to-One Sequence-to-Sequence Voice Conversion using Self-Supervised Discrete Speech Representations
arXiv:2010.12231
Abstract
We present a novel approach to any-to-one (A2O) voice conversion (VC) in a sequence-to-sequence (seq2seq) framework. A2O VC aims to convert any speaker, including those unseen during training, to a fixed target speaker. We utilize vq-wav2vec (VQW2V), a discretized self-supervised speech representation that was learned from massive unlabeled data, which is assumed to be speaker-independent and well corresponds to underlying linguistic contents. Given a training dataset of the target speaker, we extract VQW2V and acoustic features to estimate a seq2seq mapping function from the former to the latter. With the help of a pretraining method and a newly designed postprocessing technique, our model can be generalized to only 5 min of data, even outperforming the same model trained with parallel data.
arXiv commentSubmitted to ICASSP 2021
References in corpus (7)
- Sequence to Sequence Learning with Neural Networks
- Categorical Reparameterization with Gumbel-Softmax
- Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoders
- vq-wav2vec: Self-Supervised Learning of Discrete Speech Representations
- An Open source Implementation of ITU-T Recommendation P.808 with Validation
- Voice Transformer Network: Sequence-to-Sequence Voice Conversion Using Transformer with Text-to-Speech Pretraining
- Pretraining Techniques for Sequence-to-Sequence Voice Conversion