MelNet: A Generative Model for Audio in the Frequency Domain
arXiv:1906.01083
Abstract
Capturing high-level structure in audio waveforms is challenging because a single second of audio spans tens of thousands of timesteps. While long-range dependencies are difficult to model directly in the time domain, we show that they can be more tractably modelled in two-dimensional time-frequency representations such as spectrograms. By leveraging this representational advantage, in conjunction with a highly expressive probabilistic model and a multiscale generation procedure, we design a model capable of generating high-fidelity audio samples which capture structure at timescales that time-domain models have yet to achieve. We apply our model to a variety of audio generation tasks, including unconditional speech generation, music generation, and text-to-speech synthesis---showing improvements over previous approaches in both density estimates and human judgments.
References in corpus (6)
- Generating Long Sequences with Sparse Transformers
- Deep Voice: Real-time Neural Text-to-Speech
- GANSynth: Adversarial Neural Audio Synthesis
- PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modifications
- PixelSNAIL: An Improved Autoregressive Generative Model
- Parallel Multiscale Autoregressive Density Estimation