The Shape of RemiXXXes to Come: Audio Texture Synthesis with Time-frequency Scattering

Abstract

This article explains how to apply time--frequency scattering, a convolutional operator extracting modulations in the time--frequency domain at different rates and scales, to the re-synthesis and manipulation of audio textures. After implementing phase retrieval in the scattering network by gradient backpropagation, we introduce scale--rate DAFx, a class of audio transformations expressed in the domain of time--frequency scattering coefficients. One example of scale--rate DAFx is chirp rate inversion, which causes each sonic event to be locally reversed in time while leaving the arrow of time globally unchanged. Over the past two years, our work has led to the creation of four electroacoustic pieces: ``FAVN''; ``Modulator (Scattering Transform)''; ``Experimental Palimpsest''; ``Inspection''; and a remix of Lorenzo Senni's ``XAllegroX'', released by Warp Records on a vinyl entitled ``The Shape of RemiXXXes to Come''. The source code to reproduce experiments and figures is made freely available at: https://github.com/lostanlen/scattering.m. A companion website containing demos is at: https://lostanlen.com/pubs/dafx2019

arXiv comment8 pages, 3 figures. To appear in the proceedings of the International Conference on Digital Audio Effects (DAFX-19), Birmingham, UK

References in corpus (9)

• Understanding Deep Convolutional Networks
• Deep Scattering Spectrum
• Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders
• Bitwise Neural Networks
• Joint Time-Frequency Scattering for Audio Classification
• Audio Texture Synthesis with Scattering Moments
• Idealized computational models for auditory receptive fields
• Wavelet Scattering on the Pitch Spiral
• On Time-frequency Scattering and Computer Music