Spectrogram Inpainting for Interactive Generation of Instrument Sounds

Abstract

Modern approaches to sound synthesis using deep neural networks are hard to control, especially when fine-grained conditioning information is not available, hindering their adoption by musicians. In this paper, we cast the generation of individual instrumental notes as an inpainting-based task, introducing novel and unique ways to iteratively shape sounds. To this end, we propose a two-step approach: first, we adapt the VQ-VAE-2 image generation architecture to spectrograms in order to convert real-valued spectrograms into compact discrete codemaps, we then implement token-masked Transformers for the inpainting-based generation of these codemaps. We apply the proposed architecture on the NSynth dataset on masked resampling tasks. Most crucially, we open-source an interactive web interface to transform sounds by inpainting, for artists and practitioners alike, opening up to new, creative uses.

8 pages + references + appendices. 4 figures. Published as a conference paper at the The 2020 Joint Conference on AI Music Creativity, October 19-23, 2020, organized and hosted virtually by the Royal Institute of Technology (KTH), Stockholm, Sweden

References in corpus (7)

• WaveNet: A Generative Model for Raw Audio
• Parallel WaveNet: Fast High-Fidelity Speech Synthesis
• GANSynth: Adversarial Neural Audio Synthesis
• MorpheuS: generating structured music with constrained patterns and tension
• Universal audio synthesizer control with normalizing flows
• Vector Quantized Contrastive Predictive Coding for Template-based Music Generation
• NONOTO: A Model-agnostic Web Interface for Interactive Music Composition by Inpainting