MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis

Abstract

Previous works (Donahue et al., 2018a; Engel et al., 2019a) have found that generating coherent raw audio waveforms with GANs is challenging. In this paper, we show that it is possible to train GANs reliably to generate high quality coherent waveforms by introducing a set of architectural changes and simple training techniques. Subjective evaluation metric (Mean Opinion Score, or MOS) shows the effectiveness of the proposed approach for high quality mel-spectrogram inversion. To establish the generality of the proposed techniques, we show qualitative results of our model in speech synthesis, music domain translation and unconditional music synthesis. We evaluate the various components of the model through ablation studies and suggest a set of guidelines to design general purpose discriminators and generators for conditional sequence synthesis tasks. Our model is non-autoregressive, fully convolutional, with significantly fewer parameters than competing models and generalizes to unseen speakers for mel-spectrogram inversion. Our pytorch implementation runs at more than 100x faster than realtime on GTX 1080Ti GPU and more than 2x faster than real-time on CPU, without any hardware specific optimization tricks.

Cited by in corpus (164)

• HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis
• FastSpeech 2: Fast and High-Quality End-to-End Text to Speech
• Leveraging Frequency Analysis for Deep Fake Image Recognition
• A Survey on Neural Speech Synthesis
• DiffWave: A Versatile Diffusion Model for Audio Synthesis
• Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech
• Jukebox: A Generative Model for Music
• An Overview of Affective Speech Synthesis and Conversion in the Deep Learning Era
• AISHELL-3: A Multi-speaker Mandarin TTS Corpus and the Baselines
• AdaSpeech: Adaptive Text to Speech for Custom Voice
• AudioDec: An Open-source Streaming High-fidelity Neural Audio Codec
• Non-Attentive Tacotron: Robust and Controllable Neural TTS Synthesis Including Unsupervised Duration Modeling
• Hybrid Spectrogram and Waveform Source Separation
• Deep Representation Learning in Speech Processing: Challenges, Recent Advances, and Future Trends
• AccEar: Accelerometer Acoustic Eavesdropping with Unconstrained Vocabulary
• WaveFlow: A Compact Flow-based Model for Raw Audio
• GACELA -- A generative adversarial context encoder for long audio inpainting
• On Leveraging Pretrained GANs for Generation with Limited Data
• Visual-Tactile Cross-Modal Data Generation using Residue-Fusion GAN with Feature-Matching and Perceptual Losses
• Meta-TTS: Meta-Learning for Few-Shot Speaker Adaptive Text-to-Speech
• End-to-End Video-To-Speech Synthesis using Generative Adversarial Networks
• VoiceFixer: A Unified Framework for High-Fidelity Speech Restoration
• WaveGrad: Estimating Gradients for Waveform Generation
• Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech
• HiFi++: a Unified Framework for Bandwidth Extension and Speech Enhancement
• Audeo: Audio Generation for a Silent Performance Video
• End-to-End Adversarial Text-to-Speech
• Unconditional Audio Generation with Generative Adversarial Networks and Cycle Regularization
• Attack Agnostic Dataset: Towards Generalization and Stabilization of Audio DeepFake Detection
• ESPnet2-TTS: Extending the Edge of TTS Research
• An Overview of Voice Conversion and its Challenges: From Statistical Modeling to Deep Learning
• VoiceFixer: Toward General Speech Restoration with Neural Vocoder
• Universal MelGAN: A Robust Neural Vocoder for High-Fidelity Waveform Generation in Multiple Domains
• High Quality Streaming Speech Synthesis with Low, Sentence-Length-Independent Latency
• Latent-Domain Predictive Neural Speech Coding
• NU-GAN: High resolution neural upsampling with GAN
• WavThruVec: Latent speech representation as intermediate features for neural speech synthesis
• TFGAN: Time and Frequency Domain Based Generative Adversarial Network for High-fidelity Speech Synthesis
• Introducing Latent Timbre Synthesis
• DiscreTalk: Text-to-Speech as a Machine Translation Problem
• Multi-band MelGAN: Faster Waveform Generation for High-Quality Text-to-Speech
• A Spectral Energy Distance for Parallel Speech Synthesis
• HiFi-GAN: High-Fidelity Denoising and Dereverberation Based on Speech Deep Features in Adversarial Networks
• Taming Visually Guided Sound Generation
• EBEN: Extreme bandwidth extension network applied to speech signals captured with noise-resilient body-conduction microphones
• VQVC+: One-Shot Voice Conversion by Vector Quantization and U-Net architecture
• Adversarial representation learning for private speech generation
• Learning to Denoise Historical Music
• DrumGAN: Synthesis of Drum Sounds With Timbral Feature Conditioning Using Generative Adversarial Networks
• Rich Prosody Diversity Modelling with Phone-level Mixture Density Network
• Multi-Instrumentalist Net: Unsupervised Generation of Music from Body Movements
• Transformation-based Adversarial Video Prediction on Large-Scale Data
• Improving Neural Silent Speech Interface Models by Adversarial Training
• Learning to Maximize Speech Quality Directly Using MOS Prediction for Neural Text-to-Speech
• Emotional Prosody Control for Speech Generation
• End-to-End Zero-Shot Voice Conversion with Location-Variable Convolutions
• Bilateral Denoising Diffusion Models
• Controllable neural text-to-speech synthesis using intuitive prosodic features
• HooliGAN: Robust, High Quality Neural Vocoding
• Phonetic Posteriorgrams based Many-to-Many Singing Voice Conversion via Adversarial Training
• VoiceGrad: Non-Parallel Any-to-Many Voice Conversion with Annealed Langevin Dynamics
• Many-to-Many Voice Transformer Network
• SVSNet: An End-to-end Speaker Voice Similarity Assessment Model
• AdaSpeech 3: Adaptive Text to Speech for Spontaneous Style
• DeviceTTS: A Small-Footprint, Fast, Stable Network for On-Device Text-to-Speech
• PhaseAug: A Differentiable Augmentation for Speech Synthesis to Simulate One-to-Many Mapping
• ContextSpeech: Expressive and Efficient Text-to-Speech for Paragraph Reading
• EfficientTTS: An Efficient and High-Quality Text-to-Speech Architecture
• Fre-GAN: Adversarial Frequency-consistent Audio Synthesis
• Neural Synthesis of Footsteps Sound Effects with Generative Adversarial Networks
• Speech Synthesis and Control Using Differentiable DSP
• MFCCGAN: A Novel MFCC-Based Speech Synthesizer Using Adversarial Learning
• Parallel Synthesis for Autoregressive Speech Generation
• StarGAN-ZSVC: Towards Zero-Shot Voice Conversion in Low-Resource Contexts
• The 2020 ESPnet update: new features, broadened applications, performance improvements, and future plans
• Generative Models for Improved Naturalness, Intelligibility, and Voicing of Whispered Speech
• Chunked Autoregressive GAN for Conditional Waveform Synthesis
• How does a spontaneously speaking conversational agent affect user behavior?
• Data-Driven Modeling of Noise Time Series with Convolutional Generative Adversarial Networks
• VocGAN: A High-Fidelity Real-time Vocoder with a Hierarchically-nested Adversarial Network
• WaveGrad 2: Iterative Refinement for Text-to-Speech Synthesis
• CycleGAN-VC3: Examining and Improving CycleGAN-VCs for Mel-spectrogram Conversion
• GAN Vocoder: Multi-Resolution Discriminator Is All You Need
• StarGAN-VC+ASR: StarGAN-based Non-Parallel Voice Conversion Regularized by Automatic Speech Recognition
• LVCNet: Efficient Condition-Dependent Modeling Network for Waveform Generation
• UnivNet: A Neural Vocoder with Multi-Resolution Spectrogram Discriminators for High-Fidelity Waveform Generation
• Modeling speech recognition and synthesis simultaneously: Encoding and decoding lexical and sublexical semantic information into speech with no direct access to speech data
• Feasibility of Modeling Orthogonal Frequency-Division Multiplexing Communication Signals with Unsupervised Generative Adversarial Networks
• AGAIN-VC: A One-shot Voice Conversion using Activation Guidance and Adaptive Instance Normalization
• CUHK-EE Voice Cloning System for ICASSP 2021 M2VoC Challenge
• FastVC: Fast Voice Conversion with non-parallel data
• DiDiSpeech: A Large Scale Mandarin Speech Corpus
• Multi-Task Self-Supervised Pre-Training for Music Classification
• s-Transformer: Segment-Transformer for Robust Neural Speech Synthesis
• Source-Filter-Based Generative Adversarial Neural Vocoder for High Fidelity Speech Synthesis
• GAN Memory with No Forgetting
• CVC: Contrastive Learning for Non-parallel Voice Conversion
• Parallel and Flexible Sampling from Autoregressive Models via Langevin Dynamics
• DeepDrummer : Generating Drum Loops using Deep Learning and a Human in the Loop
• Improved parallel WaveGAN vocoder with perceptually weighted spectrogram loss
• Glow-WaveGAN: Learning Speech Representations from GAN-based Variational Auto-Encoder For High Fidelity Flow-based Speech Synthesis
• Axial Residual Networks for CycleGAN-based Voice Conversion
• Code-Mixed Text to Speech Synthesis under Low-Resource Constraints
• Puffin: pitch-synchronous neural waveform generation for fullband speech on modest devices
• Adversarially learning disentangled speech representations for robust multi-factor voice conversion
• LaughNet: synthesizing laughter utterances from waveform silhouettes and a single laughter example
• The Multi-speaker Multi-style Voice Cloning Challenge 2021
• Nonparallel Voice Conversion with Augmented Classifier Star Generative Adversarial Networks
• Building Bilingual and Code-Switched Voice Conversion with Limited Training Data Using Embedding Consistency Loss
• Neural Dubber: Dubbing for Videos According to Scripts
• DriftGAN: Using historical data for Unsupervised Recurring Drift Detection
• A Benchmarking Initiative for Audio-Domain Music Generation Using the Freesound Loop Dataset
• Quasi-Periodic Parallel WaveGAN Vocoder: A Non-autoregressive Pitch-dependent Dilated Convolution Model for Parametric Speech Generation
• High-Fidelity and Low-Latency Universal Neural Vocoder based on Multiband WaveRNN with Data-Driven Linear Prediction for Discrete Waveform Modeling
• Unified Source-Filter GAN: Unified Source-filter Network Based On Factorization of Quasi-Periodic Parallel WaveGAN
• Speech-to-Singing Conversion based on Boundary Equilibrium GAN
• Iterative Text-based Editing of Talking-heads Using Neural Retargeting
• FBWave: Efficient and Scalable Neural Vocoders for Streaming Text-To-Speech on the Edge
• FastSVC: Fast Cross-Domain Singing Voice Conversion with Feature-wise Linear Modulation
• MelGlow: Efficient Waveform Generative Network Based on Location-Variable Convolution
• Phase-aware music super-resolution using generative adversarial networks
• Gamma Boltzmann Machine for Simultaneously Modeling Linear- and Log-amplitude Spectra
• Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpus
• I'm Sorry for Your Loss: Spectrally-Based Audio Distances Are Bad at Pitch
• Parallel waveform synthesis based on generative adversarial networks with voicing-aware conditional discriminators
• NeuralDPS: Neural Deterministic Plus Stochastic Model with Multiband Excitation for Noise-Controllable Waveform Generation
• NWT: Towards natural audio-to-video generation with representation learning
• Universal Neural Vocoding with Parallel WaveNet
• DarkGAN: Exploiting Knowledge Distillation for Comprehensible Audio Synthesis with GANs
• Macedonian Speech Synthesis for Assistive Technology Applications
• Conditional Sound Generation Using Neural Discrete Time-Frequency Representation Learning
• On-device neural speech synthesis
• A Spectral Enabled GAN for Time Series Data Generation
• Text-to-Speech Synthesis Techniques for MIDI-to-Audio Synthesis
• Basis-MelGAN: Efficient Neural Vocoder Based on Audio Decomposition
• Rhythm-controllable Attention with High Robustness for Long Sentence Speech Synthesis
• AdaVocoder: Adaptive Vocoder for Custom Voice
• Rescribe: Authoring and Automatically Editing Audio Descriptions
• Source Mixing and Separation Robust Audio Steganography
• Real-time Speech Frequency Bandwidth Extension
• PeriodNet: A non-autoregressive waveform generation model with a structure separating periodic and aperiodic components
• Low Bit-Rate Wideband Speech Coding: A Deep Generative Model based Approach
• Improve GAN-based Neural Vocoder using Pointwise Relativistic LeastSquare GAN
• GANSpeech: Adversarial Training for High-Fidelity Multi-Speaker Speech Synthesis
• MaskCycleGAN-VC: Learning Non-parallel Voice Conversion with Filling in Frames
• Conditioning Trick for Training Stable GANs
• Improve Cross-lingual Voice Cloning Using Low-quality Code-switched Data
• Towards Universal Neural Vocoding with a Multi-band Excited WaveNet
• Relational Data Selection for Data Augmentation of Speaker-dependent Multi-band MelGAN Vocoder
• How Far Are We from Robust Voice Conversion: A Survey
• AI based Presentation Creator With Customized Audio Content Delivery
• DeepA: A Deep Neural Analyzer For Speech And Singing Vocoding
• Applying Phonological Features in Multilingual Text-To-Speech
• AdaSpeech 2: Adaptive Text to Speech with Untranscribed Data
• Information Sieve: Content Leakage Reduction in End-to-End Prosody For Expressive Speech Synthesis
• RefineGAN: Universally Generating Waveform Better than Ground Truth with Highly Accurate Pitch and Intensity Responses
• GANtron: Emotional Speech Synthesis with Generative Adversarial Networks
• Improving the expressiveness of neural vocoding with non-affine Normalizing Flows
• Generating Diverse Realistic Laughter for Interactive Art
• Style-Restricted GAN: Multi-Modal Translation with Style Restriction Using Generative Adversarial Networks
• Low-Latency Real-Time Non-Parallel Voice Conversion based on Cyclic Variational Autoencoder and Multiband WaveRNN with Data-Driven Linear Prediction
• Sprachsynthese -- State-of-the-Art in englischer und deutscher Sprache
• A Comprehensive Multi-scale Approach for Speech and Dynamics Synchrony in Talking Head Generation
• Exploring Voice Conversion based Data Augmentation in Text-Dependent Speaker Verification