ACVAE-VC: Non-parallel many-to-many voice conversion with auxiliary classifier variational autoencoder

Abstract

This paper proposes a non-parallel many-to-many voice conversion (VC) method using a variant of the conditional variational autoencoder (VAE) called an auxiliary classifier VAE (ACVAE). The proposed method has three key features. First, it adopts fully convolutional architectures to construct the encoder and decoder networks so that the networks can learn conversion rules that capture time dependencies in the acoustic feature sequences of source and target speech. Second, it uses an information-theoretic regularization for the model training to ensure that the information in the attribute class label will not be lost in the conversion process. With regular CVAEs, the encoder and decoder are free to ignore the attribute class label input. This can be problematic since in such a situation, the attribute class label will have little effect on controlling the voice characteristics of input speech at test time. Such situations can be avoided by introducing an auxiliary classifier and training the encoder and decoder so that the attribute classes of the decoder outputs are correctly predicted by the classifier. Third, it avoids producing buzzy-sounding speech at test time by simply transplanting the spectral details of the input speech into its converted version. Subjective evaluation experiments revealed that this simple method worked reasonably well in a non-parallel many-to-many speaker identity conversion task.

arXiv commentPublised in IEEE/ACM Trans. ASLP https://ieeexplore.ieee.org/abstract/document/8718381 Please also refer to our related articles: arXiv:1806.02169, arXiv:2008.12604

References in corpus (7)

• Auto-Encoding Variational Bayes
• Conditional Image Synthesis With Auxiliary Classifier GANs
• Semi-Supervised Learning with Deep Generative Models
• Autoencoding beyond pixels using a learned similarity metric
• Parallel-Data-Free Voice Conversion Using Cycle-Consistent Adversarial Networks
• The Voice Conversion Challenge 2018: Promoting Development of Parallel and Nonparallel Methods
• StarGAN-VC: Non-parallel many-to-many voice conversion with star generative adversarial networks

Cited by in corpus (23)

• AUTOVC: Zero-Shot Voice Style Transfer with Only Autoencoder Loss
• F0-consistent many-to-many non-parallel voice conversion via conditional autoencoder
• Any-to-Many Voice Conversion with Location-Relative Sequence-to-Sequence Modeling
• Deep Encoder-Decoder Models for Unsupervised Learning of Controllable Speech Synthesis
• Blow: a single-scale hyperconditioned flow for non-parallel raw-audio voice conversion
• Unsupervised Speech Decomposition via Triple Information Bottleneck
• ConvS2S-VC: Fully convolutional sequence-to-sequence voice conversion
• One-shot Voice Conversion by Separating Speaker and Content Representations with Instance Normalization
• Non-Parallel Voice Conversion with Cyclic Variational Autoencoder
• VAW-GAN for Singing Voice Conversion with Non-parallel Training Data
• Bootstrapping non-parallel voice conversion from speaker-adaptive text-to-speech
• CycleGAN-VC2: Improved CycleGAN-based Non-parallel Voice Conversion
• Unsupervised Audiovisual Synthesis via Exemplar Autoencoders
• Axial Residual Networks for CycleGAN-based Voice Conversion
• Crossmodal Voice Conversion
• Investigation of F0 conditioning and Fully Convolutional Networks in Variational Autoencoder based Voice Conversion
• Many-to-Many Voice Conversion with Out-of-Dataset Speaker Support
• GAZEV: GAN-Based Zero-Shot Voice Conversion over Non-parallel Speech Corpus
• MoEVC: A Mixture-of-experts Voice Conversion System with Sparse Gating Mechanism for Accelerating Online Computation
• Learning in your voice: Non-parallel voice conversion based on speaker consistency loss
• Fast MVAE: Joint separation and classification of mixed sources based on multichannel variational autoencoder with auxiliary classifier
• Zero-shot Singing Technique Conversion
• Deep Generative Variational Autoencoding for Replay Spoof Detection in Automatic Speaker Verification