Deep Voice: Real-time Neural Text-to-Speech

Abstract

We present Deep Voice, a production-quality text-to-speech system constructed entirely from deep neural networks. Deep Voice lays the groundwork for truly end-to-end neural speech synthesis. The system comprises five major building blocks: a segmentation model for locating phoneme boundaries, a grapheme-to-phoneme conversion model, a phoneme duration prediction model, a fundamental frequency prediction model, and an audio synthesis model. For the segmentation model, we propose a novel way of performing phoneme boundary detection with deep neural networks using connectionist temporal classification (CTC) loss. For the audio synthesis model, we implement a variant of WaveNet that requires fewer parameters and trains faster than the original. By using a neural network for each component, our system is simpler and more flexible than traditional text-to-speech systems, where each component requires laborious feature engineering and extensive domain expertise. Finally, we show that inference with our system can be performed faster than real time and describe optimized WaveNet inference kernels on both CPU and GPU that achieve up to 400x speedups over existing implementations.

arXiv commentSubmitted to ICML 2017

References in corpus (6)

• Adam: A Method for Stochastic Optimization
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
• A note on the evaluation of generative models
• Quasi-Recurrent Neural Networks
• Sequence-to-Sequence Neural Net Models for Grapheme-to-Phoneme Conversion

Cited by in corpus (73)

• An Overview of Multi-Task Learning in Deep Neural Networks
• FastSpeech: Fast, Robust and Controllable Text to Speech
• FastSpeech 2: Fast and High-Quality End-to-End Text to Speech
• Efficiently Trainable Text-to-Speech System Based on Deep Convolutional Networks with Guided Attention
• Device Placement Optimization with Reinforcement Learning
• Deep Voice 2: Multi-Speaker Neural Text-to-Speech
• AUTOVC: Zero-Shot Voice Style Transfer with Only Autoencoder Loss
• Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions
• Neural Voice Cloning with a Few Samples
• Tacotron: Towards End-to-End Speech Synthesis
• Fast Spectrogram Inversion using Multi-head Convolutional Neural Networks
• MelNet: A Generative Model for Audio in the Frequency Domain
• High Fidelity Speech Synthesis with Adversarial Networks
• Deep Voice 3: Scaling Text-to-Speech with Convolutional Sequence Learning
• An Overview of Affective Speech Synthesis and Conversion in the Deep Learning Era
• Flowtron: an Autoregressive Flow-based Generative Network for Text-to-Speech Synthesis
• AdaSpeech: Adaptive Text to Speech for Custom Voice
• Efficient Neural Audio Synthesis
• Waveform Modeling and Generation Using Hierarchical Recurrent Neural Networks for Speech Bandwidth Extension
• Transformer based Grapheme-to-Phoneme Conversion
• Generative adversarial network-based glottal waveform model for statistical parametric speech synthesis
• Blow: a single-scale hyperconditioned flow for non-parallel raw-audio voice conversion
• Hierarchical Generative Modeling for Controllable Speech Synthesis
• HiFiSinger: Towards High-Fidelity Neural Singing Voice Synthesis
• Direct Speech-to-image Translation
• End-to-End Adversarial Text-to-Speech
• KazakhTTS: An Open-Source Kazakh Text-to-Speech Synthesis Dataset
• FloWaveNet : A Generative Flow for Raw Audio
• A Light Dual-Task Neural Network for Haze Removal
• Review of end-to-end speech synthesis technology based on deep learning
• Cross-lingual Multi-speaker Text-to-speech Synthesis for Voice Cloning without Using Parallel Corpus for Unseen Speakers
• Recent Advances and Trends in Multimodal Deep Learning: A Review
• Voice Conversion with Conditional SampleRNN
• TimbreTron: A WaveNet(CycleGAN(CQT(Audio))) Pipeline for Musical Timbre Transfer
• TTS-Portuguese Corpus: a corpus for speech synthesis in Brazilian Portuguese
• Fitting New Speakers Based on a Short Untranscribed Sample
• Listening while Speaking: Speech Chain by Deep Learning
• VARA-TTS: Non-Autoregressive Text-to-Speech Synthesis based on Very Deep VAE with Residual Attention
• TalkNet: Fully-Convolutional Non-Autoregressive Speech Synthesis Model
• CROSSBOW: Scaling Deep Learning with Small Batch Sizes on Multi-GPU Servers
• Emotional Prosody Control for Speech Generation
• Generating Albums with SampleRNN to Imitate Metal, Rock, and Punk Bands
• The Deep Arbitrary Polynomial Chaos Neural Network or how Deep Artificial Neural Networks could benefit from Data-Driven Homogeneous Chaos Theory
• SNAC: Speaker-normalized affine coupling layer in flow-based architecture for zero-shot multi-speaker text-to-speech
• Deep Text-to-Speech System with Seq2Seq Model
• Variational Bi-LSTMs
• PortaSpeech: Portable and High-Quality Generative Text-to-Speech
• PVG at WASSA 2021: A Multi-Input, Multi-Task, Transformer-Based Architecture for Empathy and Distress Prediction
• End-to-End Spoken Language Translation
• TalkNet 2: Non-Autoregressive Depth-Wise Separable Convolutional Model for Speech Synthesis with Explicit Pitch and Duration Prediction
• A Variational Prosody Model for Mapping the Context-Sensitive Variation of Functional Prosodic Prototypes
• Hierarchical Sequence to Sequence Voice Conversion with Limited Data
• Parallel WaveNet conditioned on VAE latent vectors
• DPN-GAN: Inducing Periodic Activations in Generative Adversarial Networks for High-Fidelity Audio Synthesis
• Energy Consumption of Deep Generative Audio Models
• Speaking style adaptation in Text-To-Speech synthesis using Sequence-to-sequence models with attention
• Collapsed speech segment detection and suppression for WaveNet vocoder
• AlignTTS: Efficient Feed-Forward Text-to-Speech System without Explicit Alignment
• Voice Imitating Text-to-Speech Neural Networks
• Unpaired Image-to-Speech Synthesis with Multimodal Information Bottleneck
• Polyphone Disambiguation for Mandarin Chinese Using Conditional Neural Network with Multi-level Embedding Features
• Vision-Infused Deep Audio Inpainting
• Reducing over-smoothness in speech synthesis using Generative Adversarial Networks
• MASS: Multi-task Anthropomorphic Speech Synthesis Framework
• MCRM: Mother Compact Recurrent Memory
• Wasserstein GAN and Waveform Loss-based Acoustic Model Training for Multi-speaker Text-to-Speech Synthesis Systems Using a WaveNet Vocoder
• DeepA: A Deep Neural Analyzer For Speech And Singing Vocoding
• Normalization effects on shallow neural networks and related asymptotic expansions
• Do WaveNets Dream of Acoustic Waves?
• AI based Presentation Creator With Customized Audio Content Delivery
• RotLSTM: Rotating Memories in Recurrent Neural Networks
• Securing Voice-driven Interfaces against Fake (Cloned) Audio Attacks
• GANtron: Emotional Speech Synthesis with Generative Adversarial Networks