Synthesizing Diverse, High-Quality Audio Textures

Abstract

Texture synthesis techniques based on matching the Gram matrix of feature activations in neural networks have achieved spectacular success in the image domain. In this paper we extend these techniques to the audio domain. We demonstrate that synthesizing diverse audio textures is challenging, and argue that this is because audio data is relatively low-dimensional. We therefore introduce two new terms to the original Grammian loss: an autocorrelation term that preserves rhythm, and a diversity term that encourages the optimization procedure to synthesize unique textures. We quantitatively study the impact of our design choices on the quality of the synthesized audio by introducing an audio analogue to the Inception loss which we term the VGGish loss. We show that there is a trade-off between the diversity and quality of the synthesized audio using this technique. We additionally perform a number of experiments to qualitatively study how these design choices impact the quality of the synthesized audio. Finally we describe the implications of these results for the problem of audio style transfer.

arXiv comment10 pages, submitted to TASLP

References in corpus (9)

• WaveNet: A Generative Model for Raw Audio
• Improved Techniques for Training GANs
• A Learned Representation For Artistic Style
• Texture Networks: Feed-forward Synthesis of Textures and Stylized Images
• Texture Synthesis Using Convolutional Neural Networks
• Audio style transfer
• A Powerful Generative Model Using Random Weights for the Deep Image Representation
• Texture Synthesis Using Shallow Convolutional Networks with Random Filters
• Randomness in Deconvolutional Networks for Visual Representation

Cited by in corpus (1)

• Sound texture synthesis using convolutional neural networks