Deep Voice: Real-time Neural Text-to-Speech

Abstract

We present Deep Voice, a production-quality text-to-speech system constructed entirely from deep neural networks. Deep Voice lays the groundwork for truly end-to-end neural speech synthesis. The system comprises five major building blocks: a segmentation model for locating phoneme boundaries, a grapheme-to-phoneme conversion model, a phoneme duration prediction model, a fundamental frequency prediction model, and an audio synthesis model. For the segmentation model, we propose a novel way of performing phoneme boundary detection with deep neural networks using connectionist temporal classification (CTC) loss. For the audio synthesis model, we implement a variant of WaveNet that requires fewer parameters and trains faster than the original. By using a neural network for each component, our system is simpler and more flexible than traditional text-to-speech systems, where each component requires laborious feature engineering and extensive domain expertise. Finally, we show that inference with our system can be performed faster than real time and describe optimized WaveNet inference kernels on both CPU and GPU that achieve up to 400x speedups over existing implementations.

Submitted to ICML 2017

References in corpus (2)

• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Quasi-Recurrent Neural Networks

Cited by in corpus (34)

• An Overview of Multi-Task Learning in Deep Neural Networks
• Device Placement Optimization with Reinforcement Learning
• AUTOVC: Zero-Shot Voice Style Transfer with Only Autoencoder Loss
• MelNet: A Generative Model for Audio in the Frequency Domain
• High Fidelity Speech Synthesis with Adversarial Networks
• Flowtron: an Autoregressive Flow-based Generative Network for Text-to-Speech Synthesis
• AdaSpeech: Adaptive Text to Speech for Custom Voice
• Waveform Modeling and Generation Using Hierarchical Recurrent Neural Networks for Speech Bandwidth Extension
• Generative adversarial network-based glottal waveform model for statistical parametric speech synthesis
• HiFiSinger: Towards High-Fidelity Neural Singing Voice Synthesis
• Direct Speech-to-image Translation
• A Light Dual-Task Neural Network for Haze Removal
• Review of end-to-end speech synthesis technology based on deep learning
• Cross-lingual Multi-speaker Text-to-speech Synthesis for Voice Cloning without Using Parallel Corpus for Unseen Speakers
• Recent Advances and Trends in Multimodal Deep Learning: A Review
• Listening while Speaking: Speech Chain by Deep Learning
• VARA-TTS: Non-Autoregressive Text-to-Speech Synthesis based on Very Deep VAE with Residual Attention
• TalkNet: Fully-Convolutional Non-Autoregressive Speech Synthesis Model
• CROSSBOW: Scaling Deep Learning with Small Batch Sizes on Multi-GPU Servers
• Emotional Prosody Control for Speech Generation
• Deep Text-to-Speech System with Seq2Seq Model
• Variational Bi-LSTMs
• PVG at WASSA 2021: A Multi-Input, Multi-Task, Transformer-Based Architecture for Empathy and Distress Prediction
• End-to-End Spoken Language Translation
• Hierarchical Sequence to Sequence Voice Conversion with Limited Data
• AlignTTS: Efficient Feed-Forward Text-to-Speech System without Explicit Alignment
• Energy Consumption of Deep Generative Audio Models
• Parallel WaveNet conditioned on VAE latent vectors
• Vision-Infused Deep Audio Inpainting
• Polyphone Disambiguation for Mandarin Chinese Using Conditional Neural Network with Multi-level Embedding Features
• GANtron: Emotional Speech Synthesis with Generative Adversarial Networks
• DeepA: A Deep Neural Analyzer For Speech And Singing Vocoding
• AI based Presentation Creator With Customized Audio Content Delivery
• Securing Voice-driven Interfaces against Fake (Cloned) Audio Attacks