Neural Style Transfer for Audio Spectograms

Abstract

There has been fascinating work on creating artistic transformations of images by Gatys. This was revolutionary in how we can in some sense alter the 'style' of an image while generally preserving its 'content'. In our work, we present a method for creating new sounds using a similar approach, treating it as a style-transfer problem, starting from a random-noise input signal and iteratively using back-propagation to optimize the sound to conform to filter-outputs from a pre-trained neural architecture of interest. For demonstration, we investigate two different tasks, resulting in bandwidth expansion/compression, and timbral transfer from singing voice to musical instruments. A feature of our method is that a single architecture can generate these different audio-style-transfer types using the same set of parameters which otherwise require different complex hand-tuned diverse signal processing pipelines.

Appeared in 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA at the workshop for Machine Learning for Creativity and Design

Cited by in corpus (15)

• Neural Style Transfer: A Review
• Audio Transformers
• Pruning vs XNOR-Net: A Comprehensive Study of Deep Learning for Audio Classification on Edge-devices
• The Effect of Explicit Structure Encoding of Deep Neural Networks for Symbolic Music Generation
• Applying Visual Domain Style Transfer and Texture Synthesis Techniques to Audio - Insights and Challenges
• TimbreTron: A WaveNet(CycleGAN(CQT(Audio))) Pipeline for Musical Timbre Transfer
• Modulated Variational auto-Encoders for many-to-many musical timbre transfer
• Neuralogram: A Deep Neural Network Based Representation for Audio Signals
• Play as You Like: Timbre-enhanced Multi-modal Music Style Transfer
• Unsupervised Learning of Audio Perception for Robotics Applications: Learning to Project Data to T-SNE/UMAP space
• Learning to Model Aspects of Hearing Perception Using Neural Loss Functions
• A Framework for Generative and Contrastive Learning of Audio Representations
• Network Modulation Synthesis: New Algorithms for Generating Musical Audio Using Autoencoder Networks
• Voice Aging with Audio-Visual Style Transfer
• Actions Speak Louder than Listening: Evaluating Music Style Transfer based on Editing Experience