Parrotron: An End-to-End Speech-to-Speech Conversion Model and its Applications to Hearing-Impaired Speech and Speech Separation

Abstract

We describe Parrotron, an end-to-end-trained speech-to-speech conversion model that maps an input spectrogram directly to another spectrogram, without utilizing any intermediate discrete representation. The network is composed of an encoder, spectrogram and phoneme decoders, followed by a vocoder to synthesize a time-domain waveform. We demonstrate that this model can be trained to normalize speech from any speaker regardless of accent, prosody, and background noise, into the voice of a single canonical target speaker with a fixed accent and consistent articulation and prosody. We further show that this normalization model can be adapted to normalize highly atypical speech from a deaf speaker, resulting in significant improvements in intelligibility and naturalness, measured via a speech recognizer and listening tests. Finally, demonstrating the utility of this model on other speech tasks, we show that the same model architecture can be trained to perform a speech separation task

arXiv comment5 pages, submitted to Interspeech 2019

References in corpus (5)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting
• Attention-Based Models for Speech Recognition
• Listen and Translate: A Proof of Concept for End-to-End Speech-to-Text Translation
• AttS2S-VC: Sequence-to-Sequence Voice Conversion with Attention and Context Preservation Mechanisms

Cited by in corpus (12)

• AUTOVC: Zero-Shot Voice Style Transfer with Only Autoencoder Loss
• Unsupervised Speech Decomposition via Triple Information Bottleneck
• An Overview of Voice Conversion and its Challenges: From Statistical Modeling to Deep Learning
• WESPER: Zero-shot and Realtime Whisper to Normal Voice Conversion for Whisper-based Speech Interactions
• Direct speech-to-speech translation with a sequence-to-sequence model
• ConvS2S-VC: Fully convolutional sequence-to-sequence voice conversion
• Mel-spectrogram augmentation for sequence to sequence voice conversion
• Many-to-Many Voice Transformer Network
• FastS2S-VC: Streaming Non-Autoregressive Sequence-to-Sequence Voice Conversion
• Injecting Text in Self-Supervised Speech Pretraining
• Adversarially learning disentangled speech representations for robust multi-factor voice conversion
• Processing Phoneme Specific Segments for Cleft Lip and Palate Speech Enhancement