DarkGAN: Exploiting Knowledge Distillation for Comprehensible Audio Synthesis with GANs

Abstract

Generative Adversarial Networks (GANs) have achieved excellent audio synthesis quality in the last years. However, making them operable with semantically meaningful controls remains an open challenge. An obvious approach is to control the GAN by conditioning it on metadata contained in audio datasets. Unfortunately, audio datasets often lack the desired annotations, especially in the musical domain. A way to circumvent this lack of annotations is to generate them, for example, with an automatic audio-tagging system. The output probabilities of such systems (so-called "soft labels") carry rich information about the characteristics of the respective audios and can be used to distill the knowledge from a teacher model into a student model. In this work, we perform knowledge distillation from a large audio tagging system into an adversarial audio synthesizer that we call DarkGAN. Results show that DarkGAN can synthesize musical audio with acceptable quality and exhibits moderate attribute control even with out-of-distribution input conditioning. We release the code and provide audio examples on the accompanying website.

arXiv comment9 pages, 3 figures, 2 tables, accepted to ISMIR2021

References in corpus (19)

• Auto-Encoding Variational Bayes
• Distilling the Knowledge in a Neural Network
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• WaveNet: A Generative Model for Raw Audio
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Conditional Image Synthesis With Auxiliary Classifier GANs
• Do Deep Nets Really Need to be Deep?
• MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis
• Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders
• Conditional Generative Adversarial Networks for Speech Enhancement and Noise-Robust Speaker Verification
• GANSynth: Adversarial Neural Audio Synthesis
• Semi-supervised Knowledge Transfer for Deep Learning from Private Training Data
• Parallel-Data-Free Voice Conversion Using Cycle-Consistent Adversarial Networks
• MelNet: A Generative Model for Audio in the Frequency Domain
• Recurrent Neural Network Training with Dark Knowledge Transfer
• Jukebox: A Generative Model for Music
• Universal audio synthesizer control with normalizing flows
• Self-Supervised VQ-VAE for One-Shot Music Style Transfer
• Fréchet Audio Distance: A Metric for Evaluating Music Enhancement Algorithms