278 citations · 792 across the 28 of their papers we have counts for
36 papers
Language Models as Semantic Teachers: Post-Training Alignment for Medical Audio Understanding
Tsai-Ning Wang, Lin-Lin Chen, Neil Zeghidour +1
Pre-trained audio models excel at detecting acoustic patterns in auscultation sounds but often fail to grasp their clinical significance, limiting their use and performance in diag…
CaReAQA: A Cardiac and Respiratory Audio Question Answering Model for Open-Ended Diagnostic Reasoning
Tsai-Ning Wang, Lin-Lin Chen, Neil Zeghidour +1
Medical audio signals, such as heart and lung sounds, play a crucial role in clinical diagnosis. However, analyzing these signals remains challenging: traditional methods rely on h…
MAD Speech: Measures of Acoustic Diversity of Speech
Matthieu Futeral, Andrea Agostinelli, Marco Tagliasacchi +2
Generative spoken language models produce speech in a wide range of voices, prosody, and recording conditions, seemingly approaching the diversity of natural speech. However, the e…
MusicRL: Aligning Music Generation to Human Preferences
Geoffrey Cideron, Sertan Girgin, Mauro Verzetti +11
We propose MusicRL, the first music generation system finetuned from human feedback. Appreciation of text-to-music models is particularly subjective since the concept of musicality…
TokenSplit: Using Discrete Speech Representations for Direct, Refined, and Transcript-Conditioned Speech Separation and Recognition
Hakan Erdogan, Scott Wisdom, Xuankai Chang +4
We present TokenSplit, a speech separation model that acts on discrete token sequences. The model is trained on multiple tasks simultaneously: separate and transcribe each speech s…
AudioPaLM: A Large Language Model That Can Speak and Listen
Paul K. Rubenstein, Chulayuth Asawaroengchai, Duc Dung Nguyen +27
We introduce AudioPaLM, a large language model for speech understanding and generation. AudioPaLM fuses text-based and speech-based language models, PaLM-2 [Anil et al., 2023] and…
SoundStorm: Efficient Parallel Audio Generation
Zalán Borsos, Matt Sharifi, Damien Vincent +3
We present SoundStorm, a model for efficient, non-autoregressive audio generation. SoundStorm receives as input the semantic tokens of AudioLM, and relies on bidirectional attentio…
LMCodec: A Low Bitrate Speech Codec With Causal Transformer Models
Teerapat Jenrungrot, Michael Chinen, W. Bastiaan Kleijn +4
We introduce LMCodec, a causal neural speech codec that provides high quality audio at very low bitrates. The backbone of the system is a causal convolutional codec that encodes au…
Speech Intelligibility Classifiers from 550k Disordered Speech Samples
Subhashini Venugopalan, Jimmy Tobin, Samuel J. Yang +7
We developed dysarthric speech intelligibility classifiers on 551,176 disordered speech samples contributed by a diverse set of 468 speakers, with a range of self-reported speaking…
Speak, Read and Prompt: High-Fidelity Text-to-Speech with Minimal Supervision
Eugene Kharitonov, Damien Vincent, Zalán Borsos +6
We introduce SPEAR-TTS, a multi-speaker text-to-speech (TTS) system that can be trained with minimal supervision. By combining two types of discrete speech representations, we cast…
DNArch: Learning Convolutional Neural Architectures by Backpropagation
David W. Romero, Neil Zeghidour
We present Differentiable Neural Architectures (DNArch), a method that jointly learns the weights and the architecture of Convolutional Neural Networks (CNNs) by backpropagation. I…
SingSong: Generating musical accompaniments from singing
Chris Donahue, Antoine Caillon, Adam Roberts +8
We present SingSong, a system that generates instrumental music to accompany input vocals, potentially offering musicians and non-musicians alike an intuitive new way to create mus…
MusicLM: Generating Music From Text
Andrea Agostinelli, Timo I. Denk, Zalán Borsos +10
We introduce MusicLM, a model generating high-fidelity music from text descriptions such as "a calming violin melody backed by a distorted guitar riff". MusicLM casts the process o…
AudioLM: a Language Modeling Approach to Audio Generation
Zalán Borsos, Raphaël Marinier, Damien Vincent +8
We introduce AudioLM, a framework for high-quality audio generation with long-term consistency. AudioLM maps the input audio to a sequence of discrete tokens and casts audio genera…
Multi-instrument Music Synthesis with Spectrogram Diffusion
Curtis Hawthorne, Ian Simon, Adam Roberts +4
An ideal music synthesizer should be both interactive and expressive, generating high-fidelity audio in realtime for arbitrary combinations of instruments and notes. Recent neural…
Learning neural audio features without supervision
Sarthak Yadav, Neil Zeghidour
Deep audio classification, traditionally cast as training a deep neural network on top of mel-filterbanks in a supervised fashion, has recently benefited from two independent lines…
Disentangling speech from surroundings with neural embeddings
Ahmed Omran, Neil Zeghidour, Zalán Borsos +3
We present a method to separate speech signals from noisy environments in the embedding space of a neural audio codec. We introduce a new training procedure that allows our model t…
Learning strides in convolutional neural networks
Rachid Riad, Olivier Teboul, David Grangier +1
Convolutional neural networks typically contain several downsampling operators, such as strided convolutions or pooling layers, that progressively reduce the resolution of intermed…
General-purpose, long-context autoregressive modeling with Perceiver AR
Curtis Hawthorne, Andrew Jaegle, Cătălina Cangea +12
Real-world data is high-dimensional: a book, image, or musical performance can easily contain hundreds of thousands of elements even after compression. However, the most commonly u…
SoundStream: An End-to-End Neural Audio Codec
Neil Zeghidour, Alejandro Luebs, Ahmed Omran +2
We present SoundStream, a novel neural audio codec that can efficiently compress speech, music and general audio at bitrates normally targeted by speech-tailored codecs. SoundStrea…
DIVE: End-to-end Speech Diarization via Iterative Speaker Embedding
Neil Zeghidour, Olivier Teboul, David Grangier
We introduce DIVE, an end-to-end speaker diarization algorithm. Our neural algorithm presents the diarization task as an iterative process: it repeatedly builds a representation fo…
Self-Supervised Learning of Audio Representations from Permutations with Differentiable Ranking
Andrew N Carr, Quentin Berthet, Mathieu Blondel +2
Self-supervised pre-training using so-called "pretext" tasks has recently shown impressive performance across a wide range of modalities. In this work, we advance self-supervised l…
LEAF: A Learnable Frontend for Audio Classification
Neil Zeghidour, Olivier Teboul, Félix de Chaumont Quitry +1
Mel-filterbanks are fixed, engineered audio features which emulate human perception and have been used through the history of audio understanding up to today. However, their undeni…
Learning from Heterogeneous EEG Signals with Differentiable Channel Reordering
Aaqib Saeed, David Grangier, Olivier Pietquin +1
We propose CHARM, a method for training a single neural network across inconsistent input channels. Our work is motivated by Electroencephalography (EEG), where data collection pro…