GANSynth: Adversarial Neural Audio Synthesis

Abstract

Efficient audio synthesis is an inherently difficult machine learning task, as human perception is sensitive to both global structure and fine-scale waveform coherence. Autoregressive models, such as WaveNet, model local structure at the expense of global latent structure and slow iterative sampling, while Generative Adversarial Networks (GANs), have global latent conditioning and efficient parallel sampling, but struggle to generate locally-coherent audio waveforms. Herein, we demonstrate that GANs can in fact generate high-fidelity and locally-coherent audio by modeling log magnitudes and instantaneous frequencies with sufficient frequency resolution in the spectral domain. Through extensive empirical investigations on the NSynth dataset, we demonstrate that GANs are able to outperform strong WaveNet baselines on automated and human evaluation metrics, and efficiently generate audio several orders of magnitude faster than their autoregressive counterparts.

Colab Notebook: http://goo.gl/magenta/gansynth-demo

References in corpus (10)

• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• WaveNet: A Generative Model for Raw Audio
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Conditional Image Synthesis With Auxiliary Classifier GANs
• BEGAN: Boundary Equilibrium Generative Adversarial Networks
• SampleRNN: An Unconditional End-to-End Neural Audio Generation Model
• On Convergence and Stability of GANs
• Towards the Automatic Anime Characters Creation with Generative Adversarial Networks
• Adversarial Audio Synthesis
• Learning Multiscale Features Directly From Waveforms

Cited by in corpus (61)

• Deep Learning for Time Series Forecasting: Tutorial and Literature Survey
• DiffWave: A Versatile Diffusion Model for Audio Synthesis
• MelNet: A Generative Model for Audio in the Frequency Domain
• Video Background Music Generation with Controllable Music Transformer
• Generating Visually Aligned Sound from Videos
• Copyright in Generative Deep Learning
• Deep Representation Learning in Speech Processing: Challenges, Recent Advances, and Future Trends
• GANmapper: geographical data translation
• WGANSing: A Multi-Voice Singing Voice Synthesizer Based on the Wasserstein-GAN
• GACELA -- A generative adversarial context encoder for long audio inpainting
• On Leveraging Pretrained GANs for Generation with Limited Data
• WaveGrad: Estimating Gradients for Waveform Generation
• Creativity and Machine Learning: A Survey
• Unconditional Audio Generation with Generative Adversarial Networks and Cycle Regularization
• Generative Adversarial Network for Radar Signal Generation
• Stochastic Restoration of Heavily Compressed Musical Audio using Generative Adversarial Networks
• NU-GAN: High resolution neural upsampling with GAN
• Explaining Deep Neural Networks
• Adversarial Learning for Improved Onsets and Frames Music Transcription
• One-shot Voice Conversion by Separating Speaker and Content Representations with Instance Normalization
• Score and Lyrics-Free Singing Voice Generation
• CLAR: Contrastive Learning of Auditory Representations
• DrumGAN: Synthesis of Drum Sounds With Timbral Feature Conditioning Using Generative Adversarial Networks
• Multi-Instrumentalist Net: Unsupervised Generation of Music from Body Movements
• Latent Translation: Crossing Modalities by Bridging Generative Models
• Online Phase Reconstruction via DNN-based Phase Differences Estimation
• Time-Frequency Phase Retrieval for Audio -- The Effect of Transform Parameters
• A Deep Generative Model for Reordering Adjacency Matrices
• Symbolic Music Generation with Diffusion Models
• VAE-Info-cGAN: Generating Synthetic Images by Combining Pixel-level and Feature-level Geospatial Conditional Inputs
• Encoding Musical Style with Transformer Autoencoders
• Latent Space Explorations of Singing Voice Synthesis using DDSP
• CDCGen: Cross-Domain Conditional Generation via Normalizing Flows and Adversarial Training
• MFCCGAN: A Novel MFCC-Based Speech Synthesizer Using Adversarial Learning
• SeismoGen: Seismic Waveform Synthesis Using Generative Adversarial Networks
• Data-Driven Modeling of Noise Time Series with Convolutional Generative Adversarial Networks
• MP3net: coherent, minute-long music generation from raw audio with a simple convolutional GAN
• Spectrogram Inpainting for Interactive Generation of Instrument Sounds
• An Exploratory Study on Perceptual Spaces of the Singing Voice
• Deep Learning Models in Software Requirements Engineering
• Diet deep generative audio models with structured lottery
• Approximating Human Judgment of Generated Image Quality
• Feasibility of Modeling Orthogonal Frequency-Division Multiplexing Communication Signals with Unsupervised Generative Adversarial Networks
• Using Cyclic Noise as the Source Signal for Neural Source-Filter-based Speech Waveform Model
• DeepDrummer : Generating Drum Loops using Deep Learning and a Human in the Loop
• Guided Generative Adversarial Neural Network for Representation Learning and High Fidelity Audio Generation using Fewer Labelled Audio Data
• DarkGAN: Exploiting Knowledge Distillation for Comprehensible Audio Synthesis with GANs
• A Benchmarking Initiative for Audio-Domain Music Generation Using the Freesound Loop Dataset
• Likelihood Contribution based Multi-scale Architecture for Generative Flows
• SpecSinGAN: Sound Effect Variation Synthesis Using Single-Image GANs
• Text-to-Speech Synthesis Techniques for MIDI-to-Audio Synthesis
• An Overview on Generative AI at Scale with Edge-Cloud Computing
• A Survey on GAN Acceleration Using Memory Compression Technique
• Face-to-Music Translation Using a Distance-Preserving Generative Adversarial Network with an Auxiliary Discriminator
• Generating Diverse Realistic Laughter for Interactive Art
• Source coding of audio signals with a generative model
• Unified Signal Compression Using a GAN with Iterative Latent Representation Optimization
• Facial Keypoint Sequence Generation from Audio
• Perceiving Music Quality with GANs
• RL-Duet: Online Music Accompaniment Generation Using Deep Reinforcement Learning
• Unsupervised Cross-Domain Speech-to-Speech Conversion with Time-Frequency Consistency