Interactive Text-to-Speech System via Joint Style Analysis
arXiv:2002.06758
Abstract
While modern TTS technologies have made significant advancements in audio quality, there is still a lack of behavior naturalness compared to conversing with people. We propose a style-embedded TTS system that generates styled responses based on the speech query style. To achieve this, the system includes a style extraction model that extracts a style embedding from the speech query, which is then used by the TTS to produce a matching response. We faced two main challenges: 1) only a small portion of the TTS training dataset has style labels, which is needed to train a multi-style TTS that respects different style embeddings during inference. 2) The TTS system and the style extraction model have disjoint training datasets. We need consistent style labels across these two datasets so that the TTS can learn to respect the labels produced by the style extraction model during inference. To solve these, we adopted a semi-supervised approach that uses the style extraction model to create style labels for the TTS dataset and applied transfer learning to learn the style embedding jointly. Our experiment results show user preference for the styled TTS responses and demonstrate the style-embedded TTS system's capability of mimicking the speech query style.
arXiv commentAccepted by Interspeech 2020
References in corpus (5)
- WaveNet: A Generative Model for Raw Audio
- Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations
- Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesis
- Efficient Neural Audio Synthesis
- Uncovering Latent Style Factors for Expressive Speech Synthesis
Cited by in corpus (6)
- A Survey on Neural Speech Synthesis
- Expressive TTS Training with Frame and Style Reconstruction Loss
- Seen and Unseen emotional style transfer for voice conversion with a new emotional speech dataset
- Emotion controllable speech synthesis using emotion-unlabeled dataset with the assistance of cross-domain speech emotion recognition
- Multi-rate attention architecture for fast streamable Text-to-speech spectrum modeling
- FBWave: Efficient and Scalable Neural Vocoders for Streaming Text-To-Speech on the Edge