Catch-A-Waveform: Learning to Generate Audio from a Single Short Example

Abstract

Models for audio generation are typically trained on hours of recordings. Here, we illustrate that capturing the essence of an audio source is typically possible from as little as a few tens of seconds from a single training signal. Specifically, we present a GAN-based generative model that can be trained on one short audio signal from any domain (e.g. speech, music, etc.) and does not require pre-training or any other form of external supervision. Once trained, our model can generate random samples of arbitrary duration that maintain semantic similarity to the training waveform, yet exhibit new compositions of its audio primitives. This enables a long line of interesting applications, including generating new jazz improvisations or new a-cappella rap variants based on a single short example, producing coherent modifications to famous songs (e.g. adding a new verse to a Beatles song based solely on the original recording), filling-in of missing parts (inpainting), extending the bandwidth of a speech signal (super-resolution), and enhancing old recordings without access to any clean training example. We show that in all cases, no more than 20 seconds of training audio commonly suffice for our model to achieve state-of-the-art results. This is despite its complete lack of prior knowledge about the nature of audio signals in general.

References in corpus (13)

• Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks
• SampleRNN: An Unconditional End-to-End Neural Audio Generation Model
• Fast Spectrogram Inversion using Multi-head Convolutional Neural Networks
• Jukebox: A Generative Model for Music
• Adversarial Audio Synthesis
• DDSP: Differentiable Digital Signal Processing
• GACELA -- A generative adversarial context encoder for long audio inpainting
• WGANSing: A Multi-Voice Singing Voice Synthesizer Based on the Wasserstein-GAN
• Unconditional Audio Generation with Generative Adversarial Networks and Cycle Regularization
• On Filter Generalization for Music Bandwidth Extension Using Deep Neural Networks
• Audio inpainting with generative adversarial network
• Temporal FiLM: Capturing Long-Range Sequence Dependencies with Feature-Wise Modulations
• Hierarchical Patch VAE-GAN: Generating Diverse Videos from a Single Sample

Cited by in corpus (3)

• Taming Visually Guided Sound Generation
• LaughNet: synthesizing laughter utterances from waveform silhouettes and a single laughter example
• SpecSinGAN: Sound Effect Variation Synthesis Using Single-Image GANs