Machine Speech Chain with One-shot Speaker Adaptation

Abstract

In previous work, we developed a closed-loop speech chain model based on deep learning, in which the architecture enabled the automatic speech recognition (ASR) and text-to-speech synthesis (TTS) components to mutually improve their performance. This was accomplished by the two parts teaching each other using both labeled and unlabeled data. This approach could significantly improve model performance within a single-speaker speech dataset, but only a slight increase could be gained in multi-speaker tasks. Furthermore, the model is still unable to handle unseen speakers. In this paper, we present a new speech chain mechanism by integrating a speaker recognition model inside the loop. We also propose extending the capability of TTS to handle unseen speakers by implementing one-shot speaker adaptation. This enables TTS to mimic voice characteristics from one speaker to another with only a one-shot speaker sample, even from a text without any speaker information. In the speech chain loop mechanism, ASR also benefits from the ability to further learn an arbitrary speaker's characteristics from the generated speech waveform, resulting in a significant improvement in the recognition rate.

References in corpus (5)

• Sequence to Sequence Learning with Neural Networks
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• WaveNet: A Generative Model for Raw Audio
• Deep Speaker: an End-to-End Neural Speaker Embedding System
• Tacotron: Towards End-to-End Speech Synthesis

Cited by in corpus (10)

• Review of end-to-end speech synthesis technology based on deep learning
• One-shot Voice Conversion by Separating Speaker and Content Representations with Instance Normalization
• Leveraging Weakly Supervised Data to Improve End-to-End Speech-to-Text Translation
• Semi-supervised ASR by End-to-end Self-training
• Measuring the Effectiveness of Voice Conversion on Speaker Identification and Automatic Speech Recognition Systems
• End-to-End Feedback Loss in Speech Chain Framework via Straight-Through Estimator
• Injecting Text in Self-Supervised Speech Pretraining
• Exploring Machine Speech Chain for Domain Adaptation and Few-Shot Speaker Adaptation
• Multi-scale Alignment and Contextual History for Attention Mechanism in Sequence-to-sequence Model
• Listening while Speaking and Visualizing: Improving ASR through Multimodal Chain