Adversarial Audio Synthesis

Abstract

Audio signals are sampled at high temporal resolutions, and learning to synthesize audio requires capturing structure across a range of timescales. Generative adversarial networks (GANs) have seen wide success at generating images that are both locally and globally coherent, but they have seen little application to audio generation. In this paper we introduce WaveGAN, a first attempt at applying GANs to unsupervised synthesis of raw-waveform audio. WaveGAN is capable of synthesizing one second slices of audio waveforms with global coherence, suitable for sound effect generation. Our experiments demonstrate that, without labels, WaveGAN learns to produce intelligible words when trained on a small-vocabulary speech dataset, and can also synthesize audio from other domains such as drums, bird vocalizations, and piano. We compare WaveGAN to a method which applies GANs designed for image generation on image-like audio feature representations, finding both approaches to be promising.

arXiv commentPublished as a conference paper at ICLR 2019

Cited by in corpus (46)

• Efficiently Modeling Long Sequences with Structured State Spaces
• Implicit Neural Representations with Periodic Activation Functions
• GANSynth: Adversarial Neural Audio Synthesis
• A Survey on Neural Speech Synthesis
• Deepfake Video Detection Using Convolutional Vision Transformer
• DiffWave: A Versatile Diffusion Model for Audio Synthesis
• High Fidelity Speech Synthesis with Adversarial Networks
• Generating Visually Aligned Sound from Videos
• A Comprehensive Survey on Deep Music Generation: Multi-level Representations, Algorithms, Evaluations, and Future Directions
• DDSP: Differentiable Digital Signal Processing
• WGANSing: A Multi-Voice Singing Voice Synthesizer Based on the Wasserstein-GAN
• Reliable Fidelity and Diversity Metrics for Generative Models
• End-to-End Video-To-Speech Synthesis using Generative Adversarial Networks
• Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram
• Blow: a single-scale hyperconditioned flow for non-parallel raw-audio voice conversion
• CiwGAN and fiwGAN: Encoding information in acoustic data to model lexical learning with Generative Adversarial Networks
• End-to-End Adversarial Text-to-Speech
• Generative Adversarial Network for Radar Signal Generation
• Synthesis of Realistic ECG using Generative Adversarial Networks
• LakhNES: Improving multi-instrumental music generation with cross-domain pre-training
• Fast and Provable ADMM for Learning with Generative Priors
• Score and Lyrics-Free Singing Voice Generation
• Adversarial representation learning for private speech generation
• Adversarial Generation of Time-Frequency Features with application in audio synthesis
• Learning audio representations via phase prediction
• Autoencoding sensory substitution
• Conditional Generative Models for Counterfactual Explanations
• Anytime Sampling for Autoregressive Models via Ordered Autoencoding
• Speech Synthesis and Control Using Differentiable DSP
• Interpreting intermediate convolutional layers in unsupervised acoustic word classification
• MP3net: coherent, minute-long music generation from raw audio with a simple convolutional GAN
• WaveGrad 2: Iterative Refinement for Text-to-Speech Synthesis
• GAN Vocoder: Multi-Resolution Discriminator Is All You Need
• Catch-A-Waveform: Learning to Generate Audio from a Single Short Example
• A Survey on Audio Synthesis and Audio-Visual Multimodal Processing
• Identity-Based Patterns in Deep Convolutional Networks: Generative Adversarial Phonology and Reduplication
• Parallel WaveNet conditioned on VAE latent vectors
• Parallel and Flexible Sampling from Autoregressive Models via Langevin Dynamics
• A Benchmarking Initiative for Audio-Domain Music Generation Using the Freesound Loop Dataset
• GANs with Conditional Independence Graphs: On Subadditivity of Probability Divergences
• Conditional Sound Generation Using Neural Discrete Time-Frequency Representation Learning
• Training Wasserstein GANs without gradient penalties
• Generative Models for Security: Attacks, Defenses, and Opportunities
• DINO: A Conditional Energy-Based GAN for Domain Translation
• Text-to-speech for the hearing impaired
• Towards Lightweight Controllable Audio Synthesis with Conditional Implicit Neural Representations