MelNet: A Generative Model for Audio in the Frequency Domain

Abstract

Capturing high-level structure in audio waveforms is challenging because a single second of audio spans tens of thousands of timesteps. While long-range dependencies are difficult to model directly in the time domain, we show that they can be more tractably modelled in two-dimensional time-frequency representations such as spectrograms. By leveraging this representational advantage, in conjunction with a highly expressive probabilistic model and a multiscale generation procedure, we design a model capable of generating high-fidelity audio samples which capture structure at timescales that time-domain models have yet to achieve. We apply our model to a variety of audio generation tasks, including unconditional speech generation, music generation, and text-to-speech synthesis---showing improvements over previous approaches in both density estimates and human judgments.

References in corpus (22)

• Adam: A Method for Stochastic Optimization
• Neural Discrete Representation Learning
• VoxCeleb2: Deep Speaker Recognition
• Conditional Image Generation with PixelCNN Decoders
• Progressive Growing of GANs for Improved Quality, Stability, and Variation
• Generating Long Sequences with Sparse Transformers
• Deep Voice: Real-time Neural Text-to-Speech
• Grid Long Short-Term Memory
• GANSynth: Adversarial Neural Audio Synthesis
• ReNet: A Recurrent Neural Network Based Alternative to Convolutional Networks
• Tacotron: Towards End-to-End Speech Synthesis
• Fast Spectrogram Inversion using Multi-head Convolutional Neural Networks
• The challenge of realistic music generation: modelling raw audio at scale
• Sample Efficient Adaptive Text-to-Speech
• Generative Image Modeling Using Spatial LSTMs
• PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modifications
• Efficient Neural Audio Synthesis
• ClariNet: Parallel Wave Generation in End-to-End Text-to-Speech
• PixelSNAIL: An Improved Autoregressive Generative Model
• Parallel Multiscale Autoregressive Density Estimation
• Hierarchical Generative Modeling for Controllable Speech Synthesis
• Expressive Speech Synthesis via Modeling Expressions with Variational Autoencoder

Cited by in corpus (51)

• SuperNNova: an open-source framework for Bayesian, Neural Network based supernova classification
• A Survey on Neural Speech Synthesis
• DiffWave: A Versatile Diffusion Model for Audio Synthesis
• Jukebox: A Generative Model for Music
• High Fidelity Speech Synthesis with Adversarial Networks
• Any-to-Many Voice Conversion with Location-Relative Sequence-to-Sequence Modeling
• A Comprehensive Survey on Deep Music Generation: Multi-level Representations, Algorithms, Evaluations, and Future Directions
• GACELA -- A generative adversarial context encoder for long audio inpainting
• WaveGrad: Estimating Gradients for Waveform Generation
• The Ethical Implications of Generative Audio Models: A Systematic Literature Review
• End-to-End Adversarial Text-to-Speech
• Unconditional Audio Generation with Generative Adversarial Networks and Cycle Regularization
• High Quality Streaming Speech Synthesis with Low, Sentence-Length-Independent Latency
• Controllable Data Generation by Deep Learning: A Review
• Review of end-to-end speech synthesis technology based on deep learning
• Score and Lyrics-Free Singing Voice Generation
• Semi-Supervised Generative Modeling for Controllable Speech Synthesis
• VQVC+: One-Shot Voice Conversion by Vector Quantization and U-Net architecture
• DrumGAN: Synthesis of Drum Sounds With Timbral Feature Conditioning Using Generative Adversarial Networks
• Multi-Instrumentalist Net: Unsupervised Generation of Music from Body Movements
• Controllable deep melody generation via hierarchical music structure representation
• Dual Learning Music Composition and Dance Choreography
• Learning audio representations via phase prediction
• Incremental Text to Speech for Neural Sequence-to-Sequence Models using Reinforcement Learning
• Time-Frequency Phase Retrieval for Audio -- The Effect of Transform Parameters
• CSTNet: Contrastive Speech Translation Network for Self-Supervised Speech Representation Learning
• FastS2S-VC: Streaming Non-Autoregressive Sequence-to-Sequence Voice Conversion
• WaveNODE: A Continuous Normalizing Flow for Speech Synthesis
• Controllable speech synthesis by learning discrete phoneme-level prosodic representations
• Prosodic Clustering for Phoneme-level Prosody Control in End-to-End Speech Synthesis
• Phase reconstruction based on recurrent phase unwrapping with deep neural networks
• MP3net: coherent, minute-long music generation from raw audio with a simple convolutional GAN
• A Survey on Audio Synthesis and Audio-Visual Multimodal Processing
• Energy Consumption of Deep Generative Audio Models
• Synthesising Expressiveness in Peking Opera via Duration Informed Attention Network
• Learnable MFCCs for Speaker Verification
• Investigation of learning abilities on linguistic features in sequence-to-sequence text-to-speech synthesis
• DarkGAN: Exploiting Knowledge Distillation for Comprehensible Audio Synthesis with GANs
• Speech-to-Singing Conversion based on Boundary Equilibrium GAN
• Conditional Sound Generation Using Neural Discrete Time-Frequency Representation Learning
• Unsupervised Style and Content Separation by Minimizing Mutual Information for Speech Synthesis
• A Neural Lip-Sync Framework for Synthesizing Photorealistic Virtual News Anchors
• Gamma Boltzmann Machine for Simultaneously Modeling Linear- and Log-amplitude Spectra
• A Benchmarking Initiative for Audio-Domain Music Generation Using the Freesound Loop Dataset
• Generative Models for Security: Attacks, Defenses, and Opportunities
• Musical Speech: A Transformer-based Composition Tool
• Adaptive Multi-scale Detection of Acoustic Events
• Improving the expressiveness of neural vocoding with non-affine Normalizing Flows
• Modeling of Rakugo Speech and Its Limitations: Toward Speech Synthesis That Entertains Audiences
• Conditional Image Generation with One-Vs-All Classifier
• Text-to-speech for the hearing impaired