Adversarial Generation of Time-Frequency Features with application in audio synthesis
arXiv:1902.04072
Abstract
Time-frequency (TF) representations provide powerful and intuitive features for the analysis of time series such as audio. But still, generative modeling of audio in the TF domain is a subtle matter. Consequently, neural audio synthesis widely relies on directly modeling the waveform and previous attempts at unconditionally synthesizing audio from neurally generated invertible TF features still struggle to produce audio at satisfying quality. In this article, focusing on the short-time Fourier transform, we discuss the challenges that arise in audio synthesis based on generated invertible TF features and how to overcome them. We demonstrate the potential of deliberate generative TF modeling by training a generative adversarial network (GAN) on short-time Fourier features. We show that by applying our guidelines, our TF-based network was able to outperform a state-of-the-art GAN generating waveforms directly, despite the similar architecture in the two networks.
arXiv commentAccepted for publication at ICML 2019
References in corpus (15)
- GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
- WaveNet: A Generative Model for Raw Audio
- Large Scale GAN Training for High Fidelity Natural Image Synthesis
- Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition
- Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders
- SampleRNN: An Unconditional End-to-End Neural Audio Generation Model
- Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions
- A framework for invertible, real-time constant-Q transforms
- Adversarial Audio Synthesis
- End-to-end learning for music audio tagging at scale
- Characterization of Analytic Wavelet Transforms and a New Phaseless Reconstruction Algorithm
- Optimal Gabor frame bounds for separable lattices and estimates for Jacobi theta functions
- Learning Multiscale Features Directly From Waveforms
- Stability estimates for phase retrieval from discrete Gabor measurements
- Improving DNN-based Music Source Separation using Phase Features