DDSP: Differentiable Digital Signal Processing

Abstract

Most generative models of audio directly generate samples in one of two domains: time or frequency. While sufficient to express any signal, these representations are inefficient, as they do not utilize existing knowledge of how sound is generated and perceived. A third approach (vocoders/synthesizers) successfully incorporates strong domain knowledge of signal processing and perception, but has been less actively researched due to limited expressivity and difficulty integrating with modern auto-differentiation-based machine learning methods. In this paper, we introduce the Differentiable Digital Signal Processing (DDSP) library, which enables direct integration of classic signal processing elements with deep learning methods. Focusing on audio synthesis, we achieve high-fidelity generation without the need for large autoregressive models or adversarial losses, demonstrating that DDSP enables utilizing strong inductive biases without losing the expressive power of neural networks. Further, we show that combining interpretable modules permits manipulation of each separate model component, with applications such as independent control of pitch and loudness, realistic extrapolation to pitches not seen during training, blind dereverberation of room acoustics, transfer of extracted room acoustics to new environments, and transformation of timbre between disparate sources. In short, DDSP enables an interpretable and modular approach to generative modeling, without sacrificing the benefits of deep learning. The library is publicly available at https://github.com/magenta/ddsp and we welcome further contributions from the community and domain experts.

References in corpus (6)

• Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations
• SampleRNN: An Unconditional End-to-End Neural Audio Generation Model
• Fast Spectrogram Inversion using Multi-head Convolutional Neural Networks
• Adversarial Audio Synthesis
• WGANSing: A Multi-Voice Singing Voice Synthesizer Based on the Wasserstein-GAN
• Universal audio synthesizer control with normalizing flows

Cited by in corpus (33)

• FastSpeech 2: Fast and High-Quality End-to-End Text to Speech
• A Survey on Neural Speech Synthesis
• A Comprehensive Survey on Deep Music Generation: Multi-level Representations, Algorithms, Evaluations, and Future Directions
• WaveGrad: Estimating Gradients for Waveform Generation
• End-to-End Adversarial Text-to-Speech
• Unconditional Audio Generation with Generative Adversarial Networks and Cycle Regularization
• Self-Supervised VQ-VAE for One-Shot Music Style Transfer
• DrumGAN: Synthesis of Drum Sounds With Timbral Feature Conditioning Using Generative Adversarial Networks
• Neural Analysis and Synthesis: Reconstructing Speech from Self-Supervised Representations
• Real-time Denoising and Dereverberation with Tiny Recurrent U-Net
• Symbolic Music Generation with Diffusion Models
• Speech Synthesis and Control Using Differentiable DSP
• Multichannel-based learning for audio object extraction
• Latent Space Explorations of Singing Voice Synthesis using DDSP
• RAVE: A variational autoencoder for fast and high-quality neural audio synthesis
• Vector-Quantized Timbre Representation
• Catch-A-Waveform: Learning to Generate Audio from a Single Short Example
• Diet deep generative audio models with structured lottery
• Using Cyclic Noise as the Source Signal for Neural Source-Filter-based Speech Waveform Model
• A Comparison of Virtual Analog Modelling Techniques for Desktop and Embedded Implementations
• Revisiting Representation Learning for Singing Voice Separation with Sinkhorn Distances
• Real-time Timbre Transfer and Sound Synthesis using DDSP
• I'm Sorry for Your Loss: Spectrally-Based Audio Distances Are Bad at Pitch
• Generating Novel Glyph without Human Data by Learning to Communicate
• SpecSinGAN: Sound Effect Variation Synthesis Using Single-Image GANs
• Hierarchical Timbre-Painting and Articulation Generation
• Dereverberation using joint estimation of dry speech signal and acoustic system
• Conditioning Autoencoder Latent Spaces for Real-Time Timbre Interpolation and Synthesis
• End-to-End Adversarial White Box Attacks on Music Instrument Classification
• Towards Universal Neural Vocoding with a Multi-band Excited WaveNet
• Deep Learning Tools for Audacity: Helping Researchers Expand the Artist's Toolkit
• DeepA: A Deep Neural Analyzer For Speech And Singing Vocoding
• White-box Audio VST Effect Programming