Audio style transfer

Abstract

'Style transfer' among images has recently emerged as a very active research topic, fuelled by the power of convolution neural networks (CNNs), and has become fast a very popular technology in social media. This paper investigates the analogous problem in the audio domain: How to transfer the style of a reference audio signal to a target audio content? We propose a flexible framework for the task, which uses a sound texture model to extract statistics characterizing the reference audio style, followed by an optimization-based audio texture synthesis to modify the target content. In contrast to mainstream optimization-based visual transfer method, the proposed process is initialized by the target content instead of random noise and the optimized loss is only about texture, not structure. These differences proved key for audio style transfer in our experiments. In order to extract features of interest, we investigate different architectures, whether pre-trained on other tasks, as done in image style transfer, or engineered based on the human auditory system. Experimental results on different types of audio signal confirm the potential of the proposed approach.

arXiv commentICASSP 2018 - 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Apr 2018, Calgary, France. IEEE

References in corpus (11)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• WaveNet: A Generative Model for Raw Audio
• Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders
• SoundNet: Learning Sound Representations from Unlabeled Video
• Neural Style Transfer: A Review
• Deep Photo Style Transfer
• A Powerful Generative Model Using Random Weights for the Deep Image Representation
• Audio Texture Synthesis with Scattering Moments
• Texture Synthesis Using Shallow Convolutional Networks with Random Filters
• MelGAN-VC: Voice Conversion and Audio Style Transfer on arbitrarily long samples using Spectrograms
• Unifying local and non-local signal processing with graph CNNs

Cited by in corpus (15)

• Fast Spectrogram Inversion using Multi-head Convolutional Neural Networks
• Self-Supervised VQ-VAE for One-Shot Music Style Transfer
• Applying Visual Domain Style Transfer and Texture Synthesis Techniques to Audio - Insights and Challenges
• StyleTime: Style Transfer for Synthetic Time Series Generation
• Autoencoder Based Architecture For Fast & Real Time Audio Style Transfer
• Overview of Tasks and Investigation of Subjective Evaluation Methods in Environmental Sound Synthesis and Conversion
• Mask Based Unsupervised Content Transfer
• Synthesizing Diverse, High-Quality Audio Textures
• Towards a Robust Framework for Multimodal Hate Detection: A Study on Video vs. Image-based Content
• Speech-to-Singing Conversion in an Encoder-Decoder Framework
• The sound of my voice: speaker representation loss for target voice separation
• Revisiting Your Memory: Reconstruction of Affect-Contextualized Memory via EEG-guided Audiovisual Generation
• Environment Transfer for Distributed Systems
• Emotion Transfer Using Vector-Valued Infinite Task Learning
• Voice Aging with Audio-Visual Style Transfer