Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders
arXiv:1704.01279
Abstract
Generative models in vision have seen rapid progress due to algorithmic improvements and the availability of high-quality image datasets. In this paper, we offer contributions in both these areas to enable similar progress in audio modeling. First, we detail a powerful new WaveNet-style autoencoder model that conditions an autoregressive decoder on temporal codes learned from the raw audio waveform. Second, we introduce NSynth, a large-scale and high-quality dataset of musical notes that is an order of magnitude larger than comparable public datasets. Using NSynth, we demonstrate improved qualitative and quantitative performance of the WaveNet autoencoder over a well-tuned spectral autoencoder baseline. Finally, we show that the model learns a manifold of embeddings that allows for morphing between instruments, meaningfully interpolating in timbre to create new types of sounds that are realistic and expressive.
References in corpus (7)
Cited by in corpus (91)
- Deep Learning for Audio Signal Processing
- A Hierarchical Latent Vector Model for Learning Long-Term Structure in Music
- MidiNet: A Convolutional Generative Adversarial Network for Symbolic-domain Music Generation
- Neural Voice Cloning with a Few Samples
- Fast Spectrogram Inversion using Multi-head Convolutional Neural Networks
- Jukebox: A Generative Model for Music
- A Comprehensive Survey on Deep Music Generation: Multi-level Representations, Algorithms, Evaluations, and Future Directions
- A Tutorial on Deep Learning for Music Information Retrieval
- Efficient Neural Audio Synthesis
- Audio style transfer
- MIDI-VAE: Modeling Dynamics and Instrumentation of Music with Applications to Style Transfer
- BYOL for Audio: Exploring Pre-trained General-purpose Audio Representations
- Demucs: Deep Extractor for Music Sources with extra unlabeled data remixed
- GACELA -- A generative adversarial context encoder for long audio inpainting
- Your GAN is Secretly an Energy-based Model and You Should use Discriminator Driven Latent Sampling
- Blow: a single-scale hyperconditioned flow for non-parallel raw-audio voice conversion
- Generative adversarial interpolative autoencoding: adversarial training on latent space interpolations encourage convex latent distributions
- A Universal Music Translation Network
- Learning a Latent Space of Multitrack Measures
- Contrastive Learning of Musical Representations
- The Effect of Explicit Structure Encoding of Deep Neural Networks for Symbolic Music Generation
- Psychoacoustic Calibration of Loss Functions for Efficient End-to-End Neural Audio Coding
- ATST: Audio Representation Learning with Teacher-Student Transformer
- Hierarchical Autoregressive Image Models with Auxiliary Decoders
- Autoencoders for music sound modeling: a comparison of linear, shallow, deep, recurrent and variational models
- Multimodal Self-Supervised Learning of General Audio Representations
- Stochastic WaveNet: A Generative Latent Variable Model for Sequential Data
- Generative timbre spaces: regularizing variational auto-encoders with perceptual metrics
- 1-Dimensional polynomial neural networks for audio signal related problems
- TimbreTron: A WaveNet(CycleGAN(CQT(Audio))) Pipeline for Musical Timbre Transfer
- Crossing You in Style: Cross-modal Style Transfer from Music to Visual Arts
- CLAR: Contrastive Learning of Auditory Representations
- DrumGAN: Synthesis of Drum Sounds With Timbral Feature Conditioning Using Generative Adversarial Networks
- A general-purpose deep learning approach to model time-varying audio effects
- Adversarial Generation of Time-Frequency Features with application in audio synthesis
- VaPar Synth -- A Variational Parametric Model for Audio Synthesis
- Real-valued parametric conditioning of an RNN for interactive sound synthesis
- Symbolic Music Generation with Diffusion Models
- COALA: Co-Aligned Autoencoders for Learning Semantically Enriched Audio Representations
- Encoding Musical Style with Transformer Autoencoders
- Conditioning Deep Generative Raw Audio Models for Structured Automatic Music
- A comparison of recent waveform generation and acoustic modeling methods for neural-network-based speech synthesis
- A proposal and evaluation of new timbre visualisation methods for audio sample browsers
- Assisted Sound Sample Generation with Musical Conditioning in Adversarial Auto-Encoders
- Data Augmentation for Instrument Classification Robust to Audio Effects
- Rank-based loss for learning hierarchical representations
- Spectrogram Inpainting for Interactive Generation of Instrument Sounds
- An Exploratory Study on Perceptual Spaces of the Singing Voice
- Label Propagation Adaptive Resonance Theory for Semi-supervised Continuous Learning
- Learning Disentangled Representations of Timbre and Pitch for Musical Instrument Sounds Using Gaussian Mixture Variational Autoencoders
- RAVE: A variational autoencoder for fast and high-quality neural audio synthesis
- An Investigation of the Effectiveness of Phase for Audio Classification
- Multimodal Deep Learning for Mental Disorders Prediction from Audio Speech Samples
- Neural Drum Machine : An Interactive System for Real-time Synthesis of Drum Sounds
- Multi-task WaveNet: A Multi-task Generative Model for Statistical Parametric Speech Synthesis without Fundamental Frequency Conditions
- Diet deep generative audio models with structured lottery
- The Shape of RemiXXXes to Come: Audio Texture Synthesis with Time-frequency Scattering
- Message Passing Adaptive Resonance Theory for Online Active Semi-supervised Learning
- What Can Knowledge Bring to Machine Learning? -- A Survey of Low-shot Learning for Structured Data
- Using Cyclic Noise as the Source Signal for Neural Source-Filter-based Speech Waveform Model
- Vertical-Horizontal Structured Attention for Generating Music with Chords
- Modeling of nonlinear audio effects with end-to-end deep neural networks
- Towards Learning Universal Audio Representations
- Deep Sensory Substitution: Noninvasively Enabling Biological Neural Networks to Receive Input from Artificial Neural Networks
- DarkGAN: Exploiting Knowledge Distillation for Comprehensible Audio Synthesis with GANs
- Late reverberation suppression using U-nets
- Enhanced Memory Network: The novel network structure for Symbolic Music Generation
- A Benchmarking Initiative for Audio-Domain Music Generation Using the Freesound Loop Dataset
- I'm Sorry for Your Loss: Spectrally-Based Audio Distances Are Bad at Pitch
- DEEPF0: End-To-End Fundamental Frequency Estimation for Music and Speech Signals
- HpRNet : Incorporating Residual Noise Modeling for Violin in a Variational Parametric Synthesizer
- Learning to Fuse Music Genres with Generative Adversarial Dual Learning
- Hierarchical Timbre-Painting and Articulation Generation
- Neural View-Interpolation for Sparse Light Field Video
- Robot Sound Interpretation: Combining Sight and Sound in Learning-Based Control
- Neural Wavetable: a playable wavetable synthesizer using neural networks
- Transferring neural speech waveform synthesizers to musical instrument sounds generation
- Learning Signal-Agnostic Manifolds of Neural Fields
- Ultra-light deep MIR by trimming lottery tickets
- Deep Unsupervised Drum Transcription
- Speech Representations and Phoneme Classification for Preserving the Endangered Language of Ladin
- Masked Autoencoders with Multi-Window Local-Global Attention Are Better Audio Learners
- PerformanceNet: Score-to-Audio Music Generation with Multi-Band Convolutional Residual Network
- Modeling Melodic Feature Dependency with Modularized Variational Auto-Encoder
- Evaluation of Latent Space Disentanglement in the Presence of Interdependent Attributes
- Actions Speak Louder than Listening: Evaluating Music Style Transfer based on Editing Experience
- End-To-End Dilated Variational Autoencoder with Bottleneck Discriminative Loss for Sound Morphing -- A Preliminary Study
- Network Modulation Synthesis: New Algorithms for Generating Musical Audio Using Autoencoder Networks
- Automatic Feature Extraction for Heartbeat Anomaly Detection
- Towards Lightweight Controllable Audio Synthesis with Conditional Implicit Neural Representations
- An empirical study of weakly supervised audio tagging embeddings for general audio representations