SampleRNN: An Unconditional End-to-End Neural Audio Generation Model

Abstract

In this paper we propose a novel model for unconditional audio generation based on generating one audio sample at a time. We show that our model, which profits from combining memory-less modules, namely autoregressive multilayer perceptrons, and stateful recurrent neural networks in a hierarchical structure is able to capture underlying sources of variations in the temporal sequences over very long time spans, on three datasets of different nature. Human evaluation on the generated samples indicate that our model is preferred over competing models. We also show how each component of the model contributes to the exhibited performance.

arXiv commentPublished as a conference paper at ICLR 2017

Cited by in corpus (109)

• Deep Learning for Audio Signal Processing
• Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders
• GAN-Leaks: A Taxonomy of Membership Inference Attacks against Generative Models
• MidiNet: A Convolutional Generative Adversarial Network for Symbolic-domain Music Generation
• Implicit Neural Representations with Periodic Activation Functions
• GANSynth: Adversarial Neural Audio Synthesis
• Deep Voice 2: Multi-Speaker Neural Text-to-Speech
• SuperNNova: an open-source framework for Bayesian, Neural Network based supernova classification
• A Survey on Neural Speech Synthesis
• Neural Voice Cloning with a Few Samples
• Tacotron: Towards End-to-End Speech Synthesis
• DiffWave: A Versatile Diffusion Model for Audio Synthesis
• Jukebox: A Generative Model for Music
• High Fidelity Speech Synthesis with Adversarial Networks
• Deep Voice 3: Scaling Text-to-Speech with Convolutional Sequence Learning
• Generative Models for Automatic Chemical Design
• Interpretable Convolutional Filters with SincNet
• Video Background Music Generation with Controllable Music Transformer
• Generating Visually Aligned Sound from Videos
• A Comprehensive Survey on Deep Music Generation: Multi-level Representations, Algorithms, Evaluations, and Future Directions
• DDSP: Differentiable Digital Signal Processing
• A Tutorial on Deep Learning for Music Information Retrieval
• Efficient Neural Audio Synthesis
• Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models
• Waveform Modeling and Generation Using Hierarchical Recurrent Neural Networks for Speech Bandwidth Extension
• ClariNet: Parallel Wave Generation in End-to-End Text-to-Speech
• Demucs: Deep Extractor for Music Sources with extra unlabeled data remixed
• SRECG: ECG Signal Super-resolution Framework for Portable/Wearable Devices in Cardiac Arrhythmias Classification
• WaveFlow: A Compact Flow-based Model for Raw Audio
• GACELA -- A generative adversarial context encoder for long audio inpainting
• Deep Encoder-Decoder Models for Unsupervised Learning of Controllable Speech Synthesis
• Dual-path RNN: efficient long sequence modeling for time-domain single-channel speech separation
• Complex-valued neural networks for machine learning on non-stationary physical data
• Blow: a single-scale hyperconditioned flow for non-parallel raw-audio voice conversion
• WaveGrad: Estimating Gradients for Waveform Generation
• TasNet: time-domain audio separation network for real-time, single-channel speech separation
• Audeo: Audio Generation for a Silent Performance Video
• End-to-End Adversarial Text-to-Speech
• Quasi-Periodic WaveNet: An Autoregressive Raw Waveform Generative Model with Pitch-dependent Dilated Convolution Neural Network
• Listening to Sounds of Silence for Speech Denoising
• NU-GAN: High resolution neural upsampling with GAN
• Latent-Domain Predictive Neural Speech Coding
• Autoencoders for music sound modeling: a comparison of linear, shallow, deep, recurrent and variational models
• Codified audio language modeling learns useful representations for music information retrieval
• Multi-band MelGAN: Faster Waveform Generation for High-Quality Text-to-Speech
• ConvS2S-VC: Fully convolutional sequence-to-sequence voice conversion
• Taming Visually Guided Sound Generation
• Generative timbre spaces: regularizing variational auto-encoders with perceptual metrics
• TimbreTron: A WaveNet(CycleGAN(CQT(Audio))) Pipeline for Musical Timbre Transfer
• Voice Conversion with Conditional SampleRNN
• Do Neural Nets Learn Statistical Laws behind Natural Language?
• NAUTILUS: a Versatile Voice Cloning System
• PixelVAE++: Improved PixelVAE with Discrete Prior
• Generating Black Metal and Math Rock: Beyond Bach, Beethoven, and Beatles
• Multi-Instrumentalist Net: Unsupervised Generation of Music from Body Movements
• Cascaded Cross-Module Residual Learning towards Lightweight End-to-End Speech Coding
• A general-purpose deep learning approach to model time-varying audio effects
• VoiceLoop: Voice Fitting and Synthesis via a Phonological Loop
• Adversarial Generation of Time-Frequency Features with application in audio synthesis
• Cross-speaker Emotion Transfer Based on Speaker Condition Layer Normalization and Semi-Supervised Training in Text-To-Speech
• From Artificial Neural Networks to Deep Learning for Music Generation -- History, Concepts and Trends
• Real-valued parametric conditioning of an RNN for interactive sound synthesis
• A Hierarchical Recurrent Neural Network for Symbolic Melody Generation
• Generating Albums with SampleRNN to Imitate Metal, Rock, and Punk Bands
• VoiceGrad: Non-Parallel Any-to-Many Voice Conversion with Annealed Langevin Dynamics
• A Unified Speaker Adaptation Method for Speech Synthesis using Transcribed and Untranscribed Speech with Backpropagation
• Quasi-Periodic WaveNet Vocoder: A Pitch Dependent Dilated Convolution Model for Parametric Speech Generation
• Many-to-Many Voice Transformer Network
• Neural source-filter waveform models for statistical parametric speech synthesis
• SingSong: Generating musical accompaniments from singing
• Chunked Autoregressive GAN for Conditional Waveform Synthesis
• Overview of Tasks and Investigation of Subjective Evaluation Methods in Environmental Sound Synthesis and Conversion
• From Speaker Verification to Multispeaker Speech Synthesis, Deep Transfer with Feedback Constraint
• RAVE: A variational autoencoder for fast and high-quality neural audio synthesis
• Non-parallel Voice Conversion System with WaveNet Vocoder and Collapsed Speech Suppression
• Catch-A-Waveform: Learning to Generate Audio from a Single Short Example
• Collapsed speech segment detection and suppression for WaveNet vocoder
• Diet deep generative audio models with structured lottery
• Knowledge-and-Data-Driven Amplitude Spectrum Prediction for Hierarchical Neural Vocoders
• Vertical-Horizontal Structured Attention for Generating Music with Chords
• Parallel and Flexible Sampling from Autoregressive Models via Langevin Dynamics
• Multi-scale Transformer Language Models
• Voice Imitating Text-to-Speech Neural Networks
• Puffin: pitch-synchronous neural waveform generation for fullband speech on modest devices
• Source-Filter-Based Generative Adversarial Neural Vocoder for High Fidelity Speech Synthesis
• Glow-WaveGAN: Learning Speech Representations from GAN-based Variational Auto-Encoder For High Fidelity Flow-based Speech Synthesis
• Vision-Infused Deep Audio Inpainting
• Investigation of learning abilities on linguistic features in sequence-to-sequence text-to-speech synthesis
• Unified Source-Filter GAN: Unified Source-filter Network Based On Factorization of Quasi-Periodic Parallel WaveGAN
• The AS-NU System for the M2VoC Challenge
• A Benchmarking Initiative for Audio-Domain Music Generation Using the Freesound Loop Dataset
• NeuralDPS: Neural Deterministic Plus Stochastic Model with Multiband Excitation for Noise-Controllable Waveform Generation
• Distilling the Knowledge from Conditional Normalizing Flows
• On-device neural speech synthesis
• Conditional Sound Generation Using Neural Discrete Time-Frequency Representation Learning
• Fast and High-Quality Singing Voice Synthesis System based on Convolutional Neural Networks
• Quasi-Periodic Parallel WaveGAN Vocoder: A Non-autoregressive Pitch-dependent Dilated Convolution Model for Parametric Speech Generation
• I'm Sorry for Your Loss: Spectrally-Based Audio Distances Are Bad at Pitch
• Nonparallel Voice Conversion with Augmented Classifier Star Generative Adversarial Networks
• Improve GAN-based Neural Vocoder using Pointwise Relativistic LeastSquare GAN
• Transferring neural speech waveform synthesizers to musical instrument sounds generation
• Generalization of Spectrum Differential based Direct Waveform Modification for Voice Conversion
• Statistical Voice Conversion with Quasi-Periodic WaveNet Vocoder
• End-To-End Dilated Variational Autoencoder with Bottleneck Discriminative Loss for Sound Morphing -- A Preliminary Study
• SchrödingeRNN: Generative Modeling of Raw Audio as a Continuously Observed Quantum State
• Text-to-speech for the hearing impaired
• Relational Data Selection for Data Augmentation of Speaker-dependent Multi-band MelGAN Vocoder
• Network Modulation Synthesis: New Algorithms for Generating Musical Audio Using Autoencoder Networks
• Modeling Melodic Feature Dependency with Modularized Variational Auto-Encoder