Autoencoders for music sound modeling: a comparison of linear, shallow, deep, recurrent and variational models

Abstract

This study investigates the use of non-linear unsupervised dimensionality reduction techniques to compress a music dataset into a low-dimensional representation which can be used in turn for the synthesis of new sounds. We systematically compare (shallow) autoencoders (AEs), deep autoencoders (DAEs), recurrent autoencoders (with Long Short-Term Memory cells -- LSTM-AEs) and variational autoencoders (VAEs) with principal component analysis (PCA) for representing the high-resolution short-term magnitude spectrum of a large and dense dataset of music notes into a lower-dimensional vector (and then convert it back to a magnitude spectrum used for sound resynthesis). Our experiments were conducted on the publicly available multi-instrument and multi-pitch database NSynth. Interestingly and contrary to the recent literature on image processing, we can show that PCA systematically outperforms shallow AE. Only deep and recurrent architectures (DAEs and LSTM-AEs) lead to a lower reconstruction error. The optimization criterion in VAEs being the sum of the reconstruction error and a regularization term, it naturally leads to a lower reconstruction accuracy than DAEs but we show that VAEs are still able to outperform PCA while providing a low-dimensional latent space with nice "usability" properties. We also provide corresponding objective measures of perceptual audio quality (PEMO-Q scores), which generally correlate well with the reconstruction error.

arXiv commentSMC 2019

References in corpus (9)

• Adam: A Method for Stochastic Optimization
• Scikit-learn: Machine Learning in Python
• WaveNet: A Generative Model for Raw Audio
• Generative Adversarial Text to Image Synthesis
• Pixel Recurrent Neural Networks
• A Recurrent Latent Variable Model for Sequential Data
• Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders
• SampleRNN: An Unconditional End-to-End Neural Audio Generation Model
• Semi-supervised multichannel speech enhancement with variational autoencoders and non-negative matrix factorization

Cited by in corpus (5)

• Deblending galaxies with Variational Autoencoders: a joint multi-band, multi-instrument approach
• VaPar Synth -- A Variational Parametric Model for Audio Synthesis
• HpRNet : Incorporating Residual Noise Modeling for Violin in a Variational Parametric Synthesizer
• Convolutional Autoencoders for Lossy Light Field Compression
• A Model You Can Hear: Audio Identification with Playable Prototypes