Review of end-to-end speech synthesis technology based on deep learning
arXiv:2104.09995
Abstract
As an indispensable part of modern human-computer interaction system, speech synthesis technology helps users get the output of intelligent machine more easily and intuitively, thus has attracted more and more attention. Due to the limitations of high complexity and low efficiency of traditional speech synthesis technology, the current research focus is the deep learning-based end-to-end speech synthesis technology, which has more powerful modeling ability and a simpler pipeline. It mainly consists of three modules: text front-end, acoustic model, and vocoder. This paper reviews the research status of these three parts, and classifies and compares various methods according to their emphasis. Moreover, this paper also summarizes the open-source speech corpus of English, Chinese and other languages that can be used for speech synthesis tasks, and introduces some commonly used subjective and objective speech quality evaluation method. Finally, some attractive future research directions are pointed out.
References in corpus (53)
- Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
- Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
- Denoising Diffusion Probabilistic Models
- GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
- Convolutional Sequence to Sequence Learning
- Self-Attention Generative Adversarial Networks
- Deep Unsupervised Learning using Nonequilibrium Thermodynamics
- NICE: Non-linear Independent Components Estimation
- Variational Inference with Normalizing Flows
- HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis
- Equivalence of distance-based and RKHS-based statistics in hypothesis testing
- Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesis
- FastSpeech 2: Fast and High-Quality End-to-End Text to Speech
- Deep Voice: Real-time Neural Text-to-Speech
- Professor Forcing: A New Algorithm for Training Recurrent Networks
- Efficiently Trainable Text-to-Speech System Based on Deep Convolutional Networks with Guided Attention
- Variational Lossy Autoencoder
- Neural Voice Cloning with a Few Samples
- Demystifying MMD GANs
- Improved Techniques for Training Score-Based Generative Models
- DiffWave: A Versatile Diffusion Model for Audio Synthesis
- MelNet: A Generative Model for Audio in the Frequency Domain
- Reinforcement Learning Neural Turing Machines - Revised
- Forward Attention in Sequence-to-sequence Acoustic Modelling for Speech Synthesis
- AISHELL-3: A Multi-speaker Mandarin TTS Corpus and the Baselines
- Flowtron: an Autoregressive Flow-based Generative Network for Text-to-Speech Synthesis
- Almost Unsupervised Text to Speech and Automatic Speech Recognition
- Sliced Score Matching: A Scalable Approach to Density and Score Estimation
- CHiVE: Varying Prosody in Speech Synthesis with a Linguistically Driven Dynamic Hierarchical Conditional Variational Network
- Blow: a single-scale hyperconditioned flow for non-parallel raw-audio voice conversion
- Unsupervised Speech Decomposition via Triple Information Bottleneck
- End-to-End Adversarial Text-to-Speech
- The Exact Equivalence of Distance and Kernel Methods for Hypothesis Testing
- FloWaveNet : A Generative Flow for Raw Audio
- High Quality Streaming Speech Synthesis with Low, Sentence-Length-Independent Latency
- Parallel Tacotron: Non-Autoregressive and Controllable TTS
- A Spectral Energy Distance for Parallel Speech Synthesis
- JDI-T: Jointly trained Duration Informed Transformer for Text-To-Speech without Explicit Alignment
- Fitting New Speakers Based on a Short Untranscribed Sample
- Listening while Speaking: Speech Chain by Deep Learning
- Unified Mandarin TTS Front-end Based on Distilled BERT Model
- GraphSpeech: Syntax-Aware Graph Attention Network For Neural Speech Synthesis
- Maximizing Mutual Information for Tacotron
- Prosody Transfer in Neural Text to Speech Using Global Pitch and Loudness Features
- Machine Speech Chain with One-shot Speaker Adaptation
- TalkNet: Fully-Convolutional Non-Autoregressive Speech Synthesis Model
- Deep Text-to-Speech System with Seq2Seq Model
- DeviceTTS: A Small-Footprint, Fast, Stable Network for On-Device Text-to-Speech
- Effective parameter estimation methods for an ExcitNet model in generative text-to-speech systems
- Incremental Text-to-Speech Synthesis with Prefix-to-Prefix Framework
- Initial investigation of an encoder-decoder end-to-end TTS framework using marginalization of monotonic hard latent alignments
- Multi-Reference Neural TTS Stylization with Adversarial Cycle Consistency
- Exploiting Syntactic Features in a Parsed Tree to Improve End-to-End TTS