DurIAN: Duration Informed Attention Network For Multimodal Synthesis
arXiv:1909.01700
Abstract
In this paper, we present a generic and robust multimodal synthesis system that produces highly natural speech and facial expression simultaneously. The key component of this system is the Duration Informed Attention Network (DurIAN), an autoregressive model in which the alignments between the input text and the output acoustic features are inferred from a duration model. This is different from the end-to-end attention mechanism used, and accounts for various unavoidable artifacts, in existing end-to-end speech synthesis systems such as Tacotron. Furthermore, DurIAN can be used to generate high quality facial expression which can be synchronized with generated speech with/without parallel speech and face data. To improve the efficiency of speech generation, we also propose a multi-band parallel generation strategy on top of the WaveRNN model. The proposed Multi-band WaveRNN effectively reduces the total computational complexity from 9.8 to 5.5 GFLOPS, and is able to generate audio that is 6 times faster than real time on a single CPU core. We show that DurIAN could generate highly natural speech that is on par with current state of the art end-to-end systems, while at the same time avoid word skipping/repeating errors in those systems. Finally, a simple yet effective approach for fine-grained control of expressiveness of speech and facial expression is introduced.
References in corpus (2)
Cited by in corpus (36)
- Any-to-Many Voice Conversion with Location-Relative Sequence-to-Sequence Modeling
- Non-Attentive Tacotron: Robust and Controllable Neural TTS Synthesis Including Unsupervised Duration Modeling
- Universal MelGAN: A Robust Neural Vocoder for High-Fidelity Waveform Generation in Multiple Domains
- VoiceFixer: Toward General Speech Restoration with Neural Vocoder
- Fine-grained Style Modeling, Transfer and Prediction in Text-to-Speech Synthesis via Phone-Level Content-Style Disentanglement
- AdaDurIAN: Few-shot Adaptation for Neural Text-to-Speech with DurIAN
- TFGAN: Time and Frequency Domain Based Generative Adversarial Network for High-fidelity Speech Synthesis
- Multi-band MelGAN: Faster Waveform Generation for High-Quality Text-to-Speech
- Parallel Tacotron: Non-Autoregressive and Controllable TTS
- JDI-T: Jointly trained Duration Informed Transformer for Text-To-Speech without Explicit Alignment
- ByteSing: A Chinese Singing Voice Synthesis System Using Duration Allocated Encoder-Decoder Acoustic Models and WaveRNN Vocoders
- DurIAN-SC: Duration Informed Attention Network based Singing Voice Conversion System
- Learning Singing From Speech
- FeatherWave: An efficient high-fidelity neural vocoder with multi-band linear prediction
- Meta-Voice: Fast few-shot style transfer for expressive voice cloning using meta learning
- DeviceTTS: A Small-Footprint, Fast, Stable Network for On-Device Text-to-Speech
- DCCRN+: Channel-wise Subband DCCRN with SNR Estimation for Speech Enhancement
- WaveGrad 2: Iterative Refinement for Text-to-Speech Synthesis
- Word-Level Style Control for Expressive, Non-attentive Speech Synthesis
- From Speaker Verification to Multispeaker Speech Synthesis, Deep Transfer with Feedback Constraint
- CUHK-EE Voice Cloning System for ICASSP 2021 M2VoC Challenge
- Peking Opera Synthesis via Duration Informed Attention Network
- DiDiSpeech: A Large Scale Mandarin Speech Corpus
- Synthesising Expressiveness in Peking Opera via Duration Informed Attention Network
- High-Fidelity and Low-Latency Universal Neural Vocoder based on Multiband WaveRNN with Data-Driven Linear Prediction for Discrete Waveform Modeling
- Transferring Source Style in Non-Parallel Voice Conversion
- Attention Forcing for Machine Translation
- Basis-MelGAN: Efficient Neural Vocoder Based on Audio Decomposition
- Triple M: A Practical Text-to-speech Synthesis System With Multi-guidance Attention And Multi-band Multi-time LPCNet
- Learn2Sing: Target Speaker Singing Voice Synthesis by learning from a Singing Teacher
- Improve GAN-based Neural Vocoder using Pointwise Relativistic LeastSquare GAN
- EditSpeech: A Text Based Speech Editing System Using Partial Inference and Bidirectional Fusion
- Fine-grained Emotion Strength Transfer, Control and Prediction for Emotional Speech Synthesis
- Relational Data Selection for Data Augmentation of Speaker-dependent Multi-band MelGAN Vocoder
- Controllable Emotion Transfer For End-to-End Speech Synthesis
- Improving Mandarin Prosodic Structure Prediction with Multi-level Contextual Information