High Fidelity Neural Audio Compression

Abstract

We introduce a state-of-the-art real-time, high-fidelity, audio codec leveraging neural networks. It consists in a streaming encoder-decoder architecture with quantized latent space trained in an end-to-end fashion. We simplify and speed-up the training by using a single multiscale spectrogram adversary that efficiently reduces artifacts and produce high-quality samples. We introduce a novel loss balancer mechanism to stabilize training: the weight of a loss now defines the fraction of the overall gradient it should represent, thus decoupling the choice of this hyper-parameter from the typical scale of the loss. Finally, we study how lightweight Transformer models can be used to further compress the obtained representation by up to 40%, while staying faster than real time. We provide a detailed description of the key design choices of the proposed model including: training objective, architectural changes and a study of various perceptual loss functions. We present an extensive subjective evaluation (MUSHRA tests) together with an ablation study for a range of bandwidths and audio domains, including speech, noisy-reverberant speech, and music. Our approach is superior to the baselines methods across all evaluated settings, considering both 24 kHz monophonic and 48 kHz stereophonic audio. Code and models are available at github.com/facebookresearch/encodec.

arXiv commentPreprint

Cited by in corpus (21)

• SemantiCodec: An Ultra Low Bitrate Semantic Audio Codec for General Sound
• UniCATS: A Unified Context-Aware Text-to-Speech Framework with Contextual VQ-Diffusion and Vocoding
• Latent-Domain Predictive Neural Speech Coding
• TextrolSpeech: A Text Style Control Speech Corpus With Codec Language Text-to-Speech Models
• AnyEnhance: A Unified Generative Model with Prompt-Guidance and Self-Critic for Voice Enhancement
• Configurable EBEN: Extreme Bandwidth Extension Network to enhance body-conducted speech capture
• Coding Speech through Vocal Tract Kinematics
• VALL-T: Decoder-Only Generative Transducer for Robust and Decoding-Controllable Text-to-Speech
• Speech Enhancement Using Continuous Embeddings of Neural Audio Codec
• SoundSpring: Loss-Resilient Audio Transceiver with Dual-Functional Masked Language Modeling
• Universal Speech Token Learning via Low-Bitrate Neural Codec and Pretrained Representations
• EMO-Codec: An In-Depth Look at Emotion Preservation capacity of Legacy and Neural Codec Models With Subjective and Objective Evaluations
• Personalized Neural Speech Codec
• Towards Improved Objective Perceptual Audio Quality Assessment -- Part 1: A Novel Data-Driven Cognitive Model
• SMITIN: Self-Monitored Inference-Time INtervention for Generative Music Transformers
• Benchmarking Music Generation Models and Metrics via Human Preference Studies
• A Multimodal Symphony: Integrating Taste and Sound through Generative AI
• TTS-Transducer: End-to-End Speech Synthesis with Neural Transducer
• FlowMAC: Conditional Flow Matching for Audio Coding at Low Bit Rates
• Make Some Noise: Towards LLM audio reasoning and generation using sound tokens
• Code Drift: Towards Idempotent Neural Audio Codecs