A Survey on Neural Speech Synthesis

Abstract

Text to speech (TTS), or speech synthesis, which aims to synthesize intelligible and natural speech given text, is a hot research topic in speech, language, and machine learning communities and has broad applications in the industry. As the development of deep learning and artificial intelligence, neural network-based TTS has significantly improved the quality of synthesized speech in recent years. In this paper, we conduct a comprehensive survey on neural TTS, aiming to provide a good understanding of current research and future trends. We focus on the key components in neural TTS, including text analysis, acoustic models and vocoders, and several advanced topics, including fast TTS, low-resource TTS, robust TTS, expressive TTS, and adaptive TTS, etc. We further summarize resources related to TTS (e.g., datasets, opensource implementations) and discuss future research directions. This survey can serve both academic researchers and industry practitioners working on TTS.

arXiv commentA comprehensive survey on TTS, 63 pages, 18 tables, 7 figures, 457 references

References in corpus (102)

• Distilling the Knowledge in a Neural Network
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• Denoising Diffusion Probabilistic Models
• Neural Architecture Search with Reinforcement Learning
• Attention-Based Models for Speech Recognition
• Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
• NICE: Non-linear Independent Components Estimation
• Variational Inference with Normalizing Flows
• Density estimation using Real NVP
• MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis
• SampleRNN: An Unconditional End-to-End Neural Audio Generation Model
• MLS: A Large-Scale Multilingual Dataset for Speech Research
• Professor Forcing: A New Algorithm for Training Recurrent Networks
• Online and Linear-Time Attention by Enforcing Monotonic Alignments
• AISHELL-2: Transforming Mandarin ASR Research Into Industrial Scale
• Multilingual Neural Machine Translation with Knowledge Distillation
• Fast Spectrogram Inversion using Multi-head Convolutional Neural Networks
• Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech
• MelNet: A Generative Model for Audio in the Frequency Domain
• Denoising Diffusion Implicit Models
• Adversarial Audio Synthesis
• JSUT corpus: free large-scale Japanese speech corpus for end-to-end speech synthesis
• AISHELL-3: A Multi-speaker Mandarin TTS Corpus and the Baselines
• Flowtron: an Autoregressive Flow-based Generative Network for Text-to-Speech Synthesis
• AdaSpeech: Adaptive Text to Speech for Custom Voice
• Almost Unsupervised Text to Speech and Automatic Speech Recognition
• DDSP: Differentiable Digital Signal Processing
• Non-Attentive Tacotron: Robust and Controllable Neural TTS Synthesis Including Unsupervised Duration Modeling
• Sequence-to-Sequence Neural Net Models for Grapheme-to-Phoneme Conversion
• CopyCat: Many-to-Many Fine-Grained Prosody Transfer for Neural Text-to-Speech
• RNN Approaches to Text Normalization: A Challenge
• On Fast Sampling of Diffusion Probabilistic Models
• NAS-BERT: Task-Agnostic and Adaptive-Size BERT Compression with Neural Architecture Search
• Learning to Efficiently Sample from Diffusion Probabilistic Models
• Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech
• HiFiSinger: Towards High-Fidelity Neural Singing Voice Synthesis
• Unsupervised Speech Decomposition via Triple Information Bottleneck
• What comprises a good talking-head video generation?: A Survey and Benchmark
• KazakhTTS: An Open-Source Kazakh Text-to-Speech Synthesis Dataset
• MultiSpeech: Multi-Speaker Text to Speech with Transformer
• Phonological Features for 0-shot Multilingual Speech Synthesis
• Modeling Prosodic Phrasing with Multi-Task Learning in Tacotron-based TTS
• Towards Transfer Learning for End-to-End Speech Synthesis from Deep Pre-Trained Language Models
• Non-Autoregressive Neural Text-to-Speech
• High Quality Streaming Speech Synthesis with Low, Sentence-Length-Independent Latency
• Universal MelGAN: A Robust Neural Vocoder for High-Fidelity Waveform Generation in Multiple Domains
• Towards Robust Neural Vocoding for Speech Generation: A Survey
• AdaDurIAN: Few-shot Adaptation for Neural Text-to-Speech with DurIAN
• Parallel Tacotron: Non-Autoregressive and Controllable TTS
• SqueezeWave: Extremely Lightweight Vocoders for On-device Speech Synthesis
• Review of end-to-end speech synthesis technology based on deep learning
• Expressive TTS Training with Frame and Style Reconstruction Loss
• Cross-lingual Multi-speaker Text-to-speech Synthesis for Voice Cloning without Using Parallel Corpus for Unseen Speakers
• Model architectures to extrapolate emotional expressions in DNN-based text-to-speech
• Expressive Neural Voice Cloning
• Multilingual Byte2Speech Models for Scalable Low-resource Speech Synthesis
• Semi-Supervised Generative Modeling for Controllable Speech Synthesis
• Unified Mandarin TTS Front-end Based on Distilled BERT Model
• VARA-TTS: Non-Autoregressive Text-to-Speech Synthesis based on Very Deep VAE with Residual Attention
• Token-Level Ensemble Distillation for Grapheme-to-Phoneme Conversion
• GraphSpeech: Syntax-Aware Graph Attention Network For Neural Speech Synthesis
• Maximizing Mutual Information for Tacotron
• Prosody Transfer in Neural Text to Speech Using Global Pitch and Loudness Features
• TalkNet: Fully-Convolutional Non-Autoregressive Speech Synthesis Model
• Parallel Tacotron 2: A Non-Autoregressive Neural TTS Model with Differentiable Duration Modeling
• Incremental Text to Speech for Neural Sequence-to-Sequence Models using Reinforcement Learning
• AdaSpeech 3: Adaptive Text to Speech for Spontaneous Style
• Diff-TTS: A Denoising Diffusion Model for Text-to-Speech
• DeviceTTS: A Small-Footprint, Fast, Stable Network for On-Device Text-to-Speech
• Neural Harmonic-plus-Noise Waveform Model with Trainable Maximum Voice Frequency for Text-to-Speech Synthesis
• A Study of Multilingual Neural Machine Translation
• Noise Robust TTS for Low Resource Speakers using Pre-trained Model and Speech Enhancement
• Robust Sequence-to-Sequence Acoustic Modeling with Stepwise Monotonic Attention for Neural TTS
• Hi-Fi Multi-Speaker English TTS Dataset
• EfficientTTS: An Efficient and High-Quality Text-to-Speech Architecture
• Fre-GAN: Adversarial Frequency-consistent Audio Synthesis
• Interactive Text-to-Speech System via Joint Style Analysis
• PnG BERT: Augmented BERT on Phonemes and Graphemes for Neural TTS
• Initial investigation of an encoder-decoder end-to-end TTS framework using marginalization of monotonic hard latent alignments
• TalkNet 2: Non-Autoregressive Depth-Wise Separable Convolutional Model for Speech Synthesis with Explicit Pitch and Duration Prediction
• TTS-by-TTS: TTS-driven Data Augmentation for Fast and High-Quality Speech Synthesis
• GAN Vocoder: Multi-Resolution Discriminator Is All You Need
• Exploring Machine Speech Chain for Domain Adaptation and Few-Shot Speaker Adaptation
• LVCNet: Efficient Condition-Dependent Modeling Network for Waveform Generation
• DiDiSpeech: A Large Scale Mandarin Speech Corpus
• CUHK-EE Voice Cloning System for ICASSP 2021 M2VoC Challenge
• Towards Multi-Scale Style Control for Expressive Speech Synthesis
• The Multi-speaker Multi-style Voice Cloning Challenge 2021
• s-Transformer: Segment-Transformer for Robust Neural Speech Synthesis
• Glow-WaveGAN: Learning Speech Representations from GAN-based Variational Auto-Encoder For High Fidelity Flow-based Speech Synthesis
• Efficient Bidirectional Neural Machine Translation
• Universal Neural Vocoding with Parallel WaveNet
• STYLER: Style Factor Modeling with Rapidity and Robustness via Speech Decomposition for Expressive and Controllable Neural Text to Speech
• Unified Source-Filter GAN: Unified Source-filter Network Based On Factorization of Quasi-Periodic Parallel WaveGAN
• The AS-NU System for the M2VoC Challenge
• Fast DCTTS: Efficient Deep Convolutional Text-to-Speech
• Using IPA-Based Tacotron for Data Efficient Cross-Lingual Speaker Adaptation and Pronunciation Enhancement
• FeatherTTS: Robust and Efficient attention based Neural TTS
• Pretraining Strategies, Waveform Model Choice, and Acoustic Configurations for Multi-Speaker End-to-End Speech Synthesis
• Improving Performance of Seen and Unseen Speech Style Transfer in End-to-end Neural TTS
• Improve GAN-based Neural Vocoder using Pointwise Relativistic LeastSquare GAN
• Triple M: A Practical Text-to-speech Synthesis System With Multi-guidance Attention And Multi-band Multi-time LPCNet

Cited by in corpus (20)

• An Overview of Affective Speech Synthesis and Conversion in the Deep Learning Era
• Augmented Datasheets for Speech Datasets and Ethical Decision-Making
• Controllable Data Generation by Deep Learning: A Review
• StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Models
• WavThruVec: Latent speech representation as intermediate features for neural speech synthesis
• Hierarchical Emotion Prediction and Control in Text-to-Speech Synthesis
• A Universal Knowledge Model and Cognitive Architecture for Prototyping AGI
• USAT: A Universal Speaker-Adaptive Text-to-Speech Approach
• Giving Robots a Voice: Human-in-the-Loop Voice Creation and open-ended Labeling
• Neural HMMs are all you need (for high-quality attention-free TTS)
• Dual-Branch Knowledge Distillation for Noise-Robust Synthetic Speech Detection
• FastLTS: Non-Autoregressive End-to-End Unconstrained Lip-to-Speech Synthesis
• AdaSpeech 3: Adaptive Text to Speech for Spontaneous Style
• A Survey on Audio Synthesis and Audio-Visual Multimodal Processing
• fairseq S^2: A Scalable and Integrable Speech Synthesis Toolkit
• Code-Mixed Text to Speech Synthesis under Low-Resource Constraints
• DelightfulTTS: The Microsoft Speech Synthesis System for Blizzard Challenge 2021
• Phase perturbation improves channel robustness for speech spoofing countermeasures
• AdaVocoder: Adaptive Vocoder for Custom Voice
• Improving Mandarin Prosodic Structure Prediction with Multi-level Contextual Information