189 citations · 393 across the 18 of their papers we have counts for
20 papers
SpectroStream: A Versatile Neural Codec for General Audio
Yunpeng Li, Kehang Han, Brian McWilliams +2
We propose SpectroStream, a full-band multi-channel neural audio codec. Successor to the well-established SoundStream, SpectroStream extends its capability beyond 24 kHz monophonic…
Live Music Models
Lyria Team, Antoine Caillon, Brian McWilliams +33
We introduce a new class of generative models for music called live music models that produce a continuous stream of music in real-time with synchronized user control. We release M…
MusicRL: Aligning Music Generation to Human Preferences
Geoffrey Cideron, Sertan Girgin, Mauro Verzetti +11
We propose MusicRL, the first music generation system finetuned from human feedback. Appreciation of text-to-music models is particularly subjective since the concept of musicality…
TokenSplit: Using Discrete Speech Representations for Direct, Refined, and Transcript-Conditioned Speech Separation and Recognition
Hakan Erdogan, Scott Wisdom, Xuankai Chang +4
We present TokenSplit, a speech separation model that acts on discrete token sequences. The model is trained on multiple tasks simultaneously: separate and transcribe each speech s…
AudioPaLM: A Large Language Model That Can Speak and Listen
Paul K. Rubenstein, Chulayuth Asawaroengchai, Duc Dung Nguyen +27
We introduce AudioPaLM, a large language model for speech understanding and generation. AudioPaLM fuses text-based and speech-based language models, PaLM-2 [Anil et al., 2023] and…
SoundStorm: Efficient Parallel Audio Generation
Zalán Borsos, Matt Sharifi, Damien Vincent +3
We present SoundStorm, a model for efficient, non-autoregressive audio generation. SoundStorm receives as input the semantic tokens of AudioLM, and relies on bidirectional attentio…
LMCodec: A Low Bitrate Speech Codec With Causal Transformer Models
Teerapat Jenrungrot, Michael Chinen, W. Bastiaan Kleijn +4
We introduce LMCodec, a causal neural speech codec that provides high quality audio at very low bitrates. The backbone of the system is a causal convolutional codec that encodes au…
Speak, Read and Prompt: High-Fidelity Text-to-Speech with Minimal Supervision
Eugene Kharitonov, Damien Vincent, Zalán Borsos +6
We introduce SPEAR-TTS, a multi-speaker text-to-speech (TTS) system that can be trained with minimal supervision. By combining two types of discrete speech representations, we cast…
MusicLM: Generating Music From Text
Andrea Agostinelli, Timo I. Denk, Zalán Borsos +10
We introduce MusicLM, a model generating high-fidelity music from text descriptions such as "a calming violin melody backed by a distorted guitar riff". MusicLM casts the process o…
AudioLM: a Language Modeling Approach to Audio Generation
Zalán Borsos, Raphaël Marinier, Damien Vincent +8
We introduce AudioLM, a framework for high-quality audio generation with long-term consistency. AudioLM maps the input audio to a sequence of discrete tokens and casts audio genera…
SpeechPainter: Text-conditioned Speech Inpainting
Zalán Borsos, Matt Sharifi, Marco Tagliasacchi
We propose SpeechPainter, a model for filling in gaps of up to one second in speech samples by leveraging an auxiliary textual input. We demonstrate that the model performs speech…
Disentangling speech from surroundings with neural embeddings
Ahmed Omran, Neil Zeghidour, Zalán Borsos +3
We present a method to separate speech signals from noisy environments in the embedding space of a neural audio codec. We introduce a new training procedure that allows our model t…
Data Summarization via Bilevel Optimization
Zalán Borsos, Mojmír Mutný, Marco Tagliasacchi +1
The increasing availability of massive data sets poses a series of challenges for machine learning. Prominent among these is the need to learn models under hardware or human resour…
MicAugment: One-shot Microphone Style Transfer
Zalán Borsos, Yunpeng Li, Beat Gfeller +1
A crucial aspect for the successful deployment of audio-based models "in-the-wild" is the robustness to the transformations introduced by heterogeneous acquisition conditions. In t…
Semi-supervised Batch Active Learning via Bilevel Optimization
Zalán Borsos, Marco Tagliasacchi, Andreas Krause
Active learning is an effective technique for reducing the labeling cost by improving data efficiency. In this work, we propose a novel batch acquisition strategy for active learni…
Coresets via Bilevel Optimization for Continual Learning and Streaming
Zalán Borsos, Mojmír Mutný, Andreas Krause
Coresets are small data summaries that are sufficient for model training. They can be maintained online, enabling efficient handling of large data streams under resource constraint…
Transfer NAS: Knowledge Transfer between Search Spaces with Transformer Agents
Zalán Borsos, Andrey Khorlin, Andrea Gesmundo
Recent advances in Neural Architecture Search (NAS) have produced state-of-the-art architectures on several tasks. NAS shifts the efforts of human experts from developing novel arc…
Online Variance Reduction with Mixtures
Zalán Borsos, Sebastian Curi, Kfir Y. Levy +1
Adaptive importance sampling for stochastic optimization is a promising approach that offers improved convergence through variance reduction. In this work, we propose a new framewo…
Inference of the three-dimensional chromatin structure and its temporal behavior
Bianca-Cristina Cristescu, Zalán Borsos, John Lygeros +2
Understanding the three-dimensional (3D) structure of the genome is essential for elucidating vital biological processes and their links to human disease. To determine how the geno…
Online Variance Reduction for Stochastic Optimization
Zalán Borsos, Andreas Krause, Kfir Y. Levy
Modern stochastic optimization methods often rely on uniform sampling which is agnostic to the underlying characteristics of the data. This might degrade the convergence by yieldin…