MelNet: A Generative Model for Audio in the Frequency Domain

Abstract

Capturing high-level structure in audio waveforms is challenging because a single second of audio spans tens of thousands of timesteps. While long-range dependencies are difficult to model directly in the time domain, we show that they can be more tractably modelled in two-dimensional time-frequency representations such as spectrograms. By leveraging this representational advantage, in conjunction with a highly expressive probabilistic model and a multiscale generation procedure, we design a model capable of generating high-fidelity audio samples which capture structure at timescales that time-domain models have yet to achieve. We apply our model to a variety of audio generation tasks, including unconditional speech generation, music generation, and text-to-speech synthesis---showing improvements over previous approaches in both density estimates and human judgments.

References in corpus (6)

• Generating Long Sequences with Sparse Transformers
• Deep Voice: Real-time Neural Text-to-Speech
• GANSynth: Adversarial Neural Audio Synthesis
• PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modifications
• PixelSNAIL: An Improved Autoregressive Generative Model
• Parallel Multiscale Autoregressive Density Estimation

Cited by in corpus (31)

• A Survey on Neural Speech Synthesis
• Jukebox: A Generative Model for Music
• High Fidelity Speech Synthesis with Adversarial Networks
• A Comprehensive Survey on Deep Music Generation: Multi-level Representations, Algorithms, Evaluations, and Future Directions
• Review of end-to-end speech synthesis technology based on deep learning
• Semi-Supervised Generative Modeling for Controllable Speech Synthesis
• VQVC+: One-Shot Voice Conversion by Vector Quantization and U-Net architecture
• Multi-Instrumentalist Net: Unsupervised Generation of Music from Body Movements
• Controllable deep melody generation via hierarchical music structure representation
• Dual Learning Music Composition and Dance Choreography
• Time-Frequency Phase Retrieval for Audio -- The Effect of Transform Parameters
• Learning audio representations via phase prediction
• CSTNet: Contrastive Speech Translation Network for Self-Supervised Speech Representation Learning
• FastS2S-VC: Streaming Non-Autoregressive Sequence-to-Sequence Voice Conversion
• WaveNODE: A Continuous Normalizing Flow for Speech Synthesis
• Controllable speech synthesis by learning discrete phoneme-level prosodic representations
• Phase reconstruction based on recurrent phase unwrapping with deep neural networks
• MP3net: coherent, minute-long music generation from raw audio with a simple convolutional GAN
• A Survey on Audio Synthesis and Audio-Visual Multimodal Processing
• Energy Consumption of Deep Generative Audio Models
• Synthesising Expressiveness in Peking Opera via Duration Informed Attention Network
• Learnable MFCCs for Speaker Verification
• Gamma Boltzmann Machine for Simultaneously Modeling Linear- and Log-amplitude Spectra
• Speech-to-Singing Conversion based on Boundary Equilibrium GAN
• DarkGAN: Exploiting Knowledge Distillation for Comprehensible Audio Synthesis with GANs
• Conditional Sound Generation Using Neural Discrete Time-Frequency Representation Learning
• Generative Models for Security: Attacks, Defenses, and Opportunities
• Text-to-speech for the hearing impaired
• Musical Speech: A Transformer-based Composition Tool
• Conditional Image Generation with One-Vs-All Classifier
• Adaptive Multi-scale Detection of Acoustic Events