Chunked Autoregressive GAN for Conditional Waveform Synthesis

Abstract

Conditional waveform synthesis models learn a distribution of audio waveforms given conditioning such as text, mel-spectrograms, or MIDI. These systems employ deep generative models that model the waveform via either sequential (autoregressive) or parallel (non-autoregressive) sampling. Generative adversarial networks (GANs) have become a common choice for non-autoregressive waveform synthesis. However, state-of-the-art GAN-based models produce artifacts when performing mel-spectrogram inversion. In this paper, we demonstrate that these artifacts correspond with an inability for the generator to learn accurate pitch and periodicity. We show that simple pitch and periodicity conditioning is insufficient for reducing this error relative to using autoregression. We discuss the inductive bias that autoregression provides for learning the relationship between instantaneous frequency and phase, and show that this inductive bias holds even when autoregressively sampling large chunks of the waveform during each forward pass. Relative to prior state-of-the-art GAN-based models, our proposed model, Chunked Autoregressive GAN (CARGAN) reduces pitch error by 40-60%, reduces training time by 58%, maintains a fast generation speed suitable for real-time or interactive applications, and maintains or improves subjective quality.

Published as a conference paper at ICLR 2022

References in corpus (13)

• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• NICE: Non-linear Independent Components Estimation
• To prune, or not to prune: exploring the efficacy of pruning for model compression
• MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis
• Generating Long Sequences with Sparse Transformers
• SampleRNN: An Unconditional End-to-End Neural Audio Generation Model
• Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech
• Jukebox: A Generative Model for Music
• High Fidelity Speech Synthesis with Adversarial Networks
• Block-Sparse Recurrent Neural Networks
• Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models
• You Only Need Adversarial Supervision for Semantic Image Synthesis