Incremental Text to Speech for Neural Sequence-to-Sequence Models using Reinforcement Learning
arXiv:2008.03096 · doi:10.21437/Interspeech.2020-1822
Abstract
Modern approaches to text to speech require the entire input character sequence to be processed before any audio is synthesised. This latency limits the suitability of such models for time-sensitive tasks like simultaneous interpretation. Interleaving the action of reading a character with that of synthesising audio reduces this latency. However, the order of this sequence of interleaved actions varies across sentences, which raises the question of how the actions should be chosen. We propose a reinforcement learning based framework to train an agent to make this decision. We compare our performance against that of deterministic, rule-based systems. Our results demonstrate that our agent successfully balances the trade-off between the latency of audio generation and the quality of synthesised audio. More broadly, we show that neural sequence-to-sequence models can be adapted to run in an incremental manner.
arXiv commentTo be published in Interspeech 2020. 5 pages, 4 figures
References in corpus (13)
- Adam: A Method for Stochastic Optimization
- Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
- Sequence Transduction with Recurrent Neural Networks
- Online and Linear-Time Attention by Enforcing Monotonic Alignments
- Neural Variational Inference and Learning in Belief Networks
- Can neural machine translation do simultaneous translation?
- MelNet: A Generative Model for Audio in the Frequency Domain
- Reinforcement Learning Neural Turing Machines - Revised
- Local Monotonic Attention Mechanism for End-to-End Speech and Language Processing
- A Neural Transducer
- STACL: Simultaneous Translation with Implicit Anticipation and Controllable Latency using Prefix-to-Prefix Framework
- Incremental Text-to-Speech Synthesis with Prefix-to-Prefix Framework
- Perceptual audio loss function for deep learning