Learning audio representations via phase prediction

Abstract

We learn audio representations by solving a novel self-supervised learning task, which consists of predicting the phase of the short-time Fourier transform from its magnitude. A convolutional encoder is used to map the magnitude spectrum of the input waveform to a lower dimensional embedding. A convolutional decoder is then used to predict the instantaneous frequency (i.e., the temporal rate of change of the phase) from such embedding. To evaluate the quality of the learned representations, we evaluate how they transfer to a wide variety of downstream audio tasks. Our experiments reveal that the phase prediction task leads to representations that generalize across different tasks, partially bridging the gap with fully-supervised models. In addition, we show that the predicted phase can be used as initialization of the Griffin-Lim algorithm, thus reducing the number of iterations needed to reconstruct the waveform in the time domain.

Submitted to ICASSP 2020

References in corpus (6)

• Efficient Estimation of Word Representations in Vector Space
• Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition
• Automatic acoustic detection of birds through deep learning: the first Bird Audio Detection challenge
• MelNet: A Generative Model for Audio in the Frequency Domain
• Adversarial Audio Synthesis
• Unsupervised Feature Learning Based on Deep Models for Environmental Audio Tagging

Cited by in corpus (5)

• Self-Supervised Speech Representation Learning: A Review
• TERA: Self-Supervised Learning of Transformer Encoder Representation for Speech
• Does Visual Self-Supervision Improve Learning of Speech Representations for Emotion Recognition?
• MP3net: coherent, minute-long music generation from raw audio with a simple convolutional GAN
• Word-level Embeddings for Cross-Task Transfer Learning in Speech Processing