Unsupervised Cross-Domain Speech-to-Speech Conversion with Time-Frequency Consistency

Abstract

In recent years generative adversarial network (GAN) based models have been successfully applied for unsupervised speech-to-speech conversion.The rich compact harmonic view of the magnitude spectrogram is considered a suitable choice for training these models with audio data. To reconstruct the speech signal first a magnitude spectrogram is generated by the neural network, which is then utilized by methods like the Griffin-Lim algorithm to reconstruct a phase spectrogram. This procedure bears the problem that the generated magnitude spectrogram may not be consistent, which is required for finding a phase such that the full spectrogram has a natural-sounding speech waveform. In this work, we approach this problem by proposing a condition encouraging spectrogram consistency during the adversarial training procedure. We demonstrate our approach on the task of translating the voice of a male speaker to that of a female speaker, and vice versa. Our experimental results on the Librispeech corpus show that the model trained with the TF consistency provides a perceptually better quality of speech-to-speech conversion.

References in corpus (6)

• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Progressive Growing of GANs for Improved Quality, Stability, and Variation
• Attentive Statistics Pooling for Deep Speaker Embedding
• GANSynth: Adversarial Neural Audio Synthesis
• Parallel-Data-Free Voice Conversion Using Cycle-Consistent Adversarial Networks
• Voice Conversion Based on Cross-Domain Features Using Variational Auto Encoders