HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis

Abstract

Several recent work on speech synthesis have employed generative adversarial networks (GANs) to produce raw waveforms. Although such methods improve the sampling efficiency and memory usage, their sample quality has not yet reached that of autoregressive and flow-based generative models. In this work, we propose HiFi-GAN, which achieves both efficient and high-fidelity speech synthesis. As speech audio consists of sinusoidal signals with various periods, we demonstrate that modeling periodic patterns of an audio is crucial for enhancing sample quality. A subjective human evaluation (mean opinion score, MOS) of a single speaker dataset indicates that our proposed method demonstrates similarity to human quality while generating 22.05 kHz high-fidelity audio 167.9 times faster than real-time on a single V100 GPU. We further show the generality of HiFi-GAN to the mel-spectrogram inversion of unseen speakers and end-to-end speech synthesis. Finally, a small footprint version of HiFi-GAN generates samples 13.4 times faster than real-time on CPU with comparable quality to an autoregressive counterpart.

arXiv commentNeurIPS 2020. Code available at https://github.com/jik876/hifi-gan

References in corpus (8)

• Decoupled Weight Decay Regularization
• Spectral Normalization for Generative Adversarial Networks
• MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis
• Deep multi-scale video prediction beyond mean square error
• High Fidelity Speech Synthesis with Adversarial Networks
• Deep Voice 3: Scaling Text-to-Speech with Convolutional Sequence Learning
• ClariNet: Parallel Wave Generation in End-to-End Text-to-Speech
• SqueezeWave: Extremely Lightweight Vocoders for On-device Speech Synthesis

Cited by in corpus (68)

• StoRM: A Diffusion-based Stochastic Regeneration Model for Speech Enhancement and Dereverberation
• A Comparison of Discrete and Soft Speech Units for Improved Voice Conversion
• An Overview of Affective Speech Synthesis and Conversion in the Deep Learning Era
• AudioDec: An Open-source Streaming High-fidelity Neural Audio Codec
• VoiceFixer: A Unified Framework for High-Fidelity Speech Restoration
• Neural Vocoder is All You Need for Speech Super-resolution
• VQTTS: High-Fidelity Text-to-Speech Synthesis with Self-Supervised VQ Acoustic Feature
• HiFi++: a Unified Framework for Bandwidth Extension and Speech Enhancement
• The Ethical Implications of Generative Audio Models: A Systematic Literature Review
• Open Challenges in Synthetic Speech Detection
• Attack Agnostic Dataset: Towards Generalization and Stabilization of Audio DeepFake Detection
• An Initial Investigation for Detecting Vocoder Fingerprints of Fake Audio
• ESPnet2-TTS: Extending the Edge of TTS Research
• Transfer Learning Framework for Low-Resource Text-to-Speech using a Large-Scale Unlabeled Speech Corpus
• StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Models
• UniCATS: A Unified Context-Aware Text-to-Speech Framework with Contextual VQ-Diffusion and Vocoding
• WESPER: Zero-shot and Realtime Whisper to Normal Voice Conversion for Whisper-based Speech Interactions
• Latent-Domain Predictive Neural Speech Coding
• WavThruVec: Latent speech representation as intermediate features for neural speech synthesis
• Daft-Exprt: Cross-Speaker Prosody Transfer on Any Text for Expressive Speech Synthesis
• Review of end-to-end speech synthesis technology based on deep learning
• FlowMur: A Stealthy and Practical Audio Backdoor Attack with Limited Knowledge
• Hierarchical Emotion Prediction and Control in Text-to-Speech Synthesis
• EBEN: Extreme bandwidth extension network applied to speech signals captured with noise-resilient body-conduction microphones
• Adversarial speech for voice privacy protection from Personalized Speech generation
• LightGrad: Lightweight Diffusion Probabilistic Model for Text-to-Speech
• Autovocoder: Fast Waveform Generation from a Learned Speech Representation using Differentiable Digital Signal Processing
• End-to-End Zero-Shot Voice Conversion with Location-Variable Convolutions
• ControlVC: Zero-Shot Voice Conversion with Time-Varying Controls on Pitch and Speed
• Direct Text to Speech Translation System using Acoustic Units
• Zero-shot text-to-speech synthesis conditioned using self-supervised speech representation model
• FastLTS: Non-Autoregressive End-to-End Unconstrained Lip-to-Speech Synthesis
• Diff-TTS: A Denoising Diffusion Model for Text-to-Speech
• Unsupervised TTS Acoustic Modeling for TTS with Conditional Disentangled Sequential VAE
• PhaseAug: A Differentiable Augmentation for Speech Synthesis to Simulate One-to-Many Mapping
• OverFlow: Putting flows on top of neural transducers for better TTS
• Mitigating Unauthorized Speech Synthesis for Voice Protection
• Hi-Fi Multi-Speaker English TTS Dataset
• ContextSpeech: Expressive and Efficient Text-to-Speech for Paragraph Reading
• Parallel Synthesis for Autoregressive Speech Generation
• SEFGAN: Harvesting the Power of Normalizing Flows and GANs for Efficient High-Quality Speech Enhancement
• Attention-based Interactive Disentangling Network for Instance-level Emotional Voice Conversion
• Likelihood Training of Schrödinger Bridge using Forward-Backward SDEs Theory
• Chunked Autoregressive GAN for Conditional Waveform Synthesis
• Neural Synthesis of Footsteps Sound Effects with Generative Adversarial Networks
• Deepfake Detection System for the ADD Challenge Track 3.2 Based on Score Fusion
• TalkNet 2: Non-Autoregressive Depth-Wise Separable Convolutional Model for Speech Synthesis with Explicit Pitch and Duration Prediction
• Anonymization of Voices in Spaces for Civic Dialogue: Measuring Impact on Empathy, Trust, and Feeling Heard
• Articulation GAN: Unsupervised modeling of articulatory learning
• CUHK-EE Voice Cloning System for ICASSP 2021 M2VoC Challenge
• Modeling speech recognition and synthesis simultaneously: Encoding and decoding lexical and sublexical semantic information into speech with no direct access to speech data
• Puffin: pitch-synchronous neural waveform generation for fullband speech on modest devices
• Revisiting IPA-based Cross-lingual Text-to-speech
• Code-Mixed Text to Speech Synthesis under Low-Resource Constraints
• TranSentence: Speech-to-speech Translation via Language-agnostic Sentence-level Speech Encoding without Language-parallel Data
• Speech Rhythm-Based Speaker Embeddings Extraction from Phonemes and Phoneme Duration for Multi-Speaker Speech Synthesis
• Quantifying Source Speaker Leakage in One-to-One Voice Conversion
• Controllable Generation of Artificial Speaker Embeddings through Discovery of Principal Directions
• Varianceflow: High-Quality and Controllable Text-to-Speech using Variance Information via Normalizing Flow
• AdaVocoder: Adaptive Vocoder for Custom Voice
• EditSpeech: A Text Based Speech Editing System Using Partial Inference and Bidirectional Fusion
• Adversarial Learning of Intermediate Acoustic Feature for End-to-End Lightweight Text-to-Speech
• AnyoneNet: Synchronized Speech and Talking Head Generation for Arbitrary Person
• Improving Mandarin Prosodic Structure Prediction with Multi-level Contextual Information
• Mathematical Vocoder Algorithm : Modified Spectral Inversion for Efficient Neural Speech Synthesis
• Pruning Self-Attention for Zero-Shot Multi-Speaker Text-to-Speech
• Adapting TTS models For New Speakers using Transfer Learning
• Improving the expressiveness of neural vocoding with non-affine Normalizing Flows