DarkGAN: Exploiting Knowledge Distillation for Comprehensible Audio Synthesis with GANs

Abstract

Generative Adversarial Networks (GANs) have achieved excellent audio synthesis quality in the last years. However, making them operable with semantically meaningful controls remains an open challenge. An obvious approach is to control the GAN by conditioning it on metadata contained in audio datasets. Unfortunately, audio datasets often lack the desired annotations, especially in the musical domain. A way to circumvent this lack of annotations is to generate them, for example, with an automatic audio-tagging system. The output probabilities of such systems (so-called "soft labels") carry rich information about the characteristics of the respective audios and can be used to distill the knowledge from a teacher model into a student model. In this work, we perform knowledge distillation from a large audio tagging system into an adversarial audio synthesizer that we call DarkGAN. Results show that DarkGAN can synthesize musical audio with acceptable quality and exhibits moderate attribute control even with out-of-distribution input conditioning. We release the code and provide audio examples on the accompanying website.

9 pages, 3 figures, 2 tables, accepted to ISMIR2021

References in corpus (11)

• Distilling the Knowledge in a Neural Network
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• WaveNet: A Generative Model for Raw Audio
• MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis
• GANSynth: Adversarial Neural Audio Synthesis
• Semi-supervised Knowledge Transfer for Deep Learning from Private Training Data
• Parallel-Data-Free Voice Conversion Using Cycle-Consistent Adversarial Networks
• MelNet: A Generative Model for Audio in the Frequency Domain
• Jukebox: A Generative Model for Music
• Universal audio synthesizer control with normalizing flows
• Fréchet Audio Distance: A Metric for Evaluating Music Enhancement Algorithms