44 citations · 129 across the 46 of their papers we have counts for
51 papers
Exploring Representation Autoencoders for Text-to-Audio Generation
Bernardo V. Miranda, Hugo Malard, Gaël Richard +2
Representation autoencoders (RAE), created by pairing a frozen pretrained self-supervised learning (SSL) encoder with a trained decoder, have recently demonstrated state-of-the-art…
NemotronLabs VoiceChat: An Open Full-duplex Speech-to-Speech Model with Tool Calling Capabilities
Jagadeesh Balam, Travis Bartley, Edresson Casanova +52
We introduce NemotronLabs VoiceChat, an open full-duplex speech-to-speech model with native tool-calling capabilities. NemotronLabs VoiceChat combines a streaming speech encoder an…
A frontend-backend architecture for tool calls in full-duplex speech models
Ke Hu, Slyne Deng, Chen Chen +10
Full-duplex speech-to-speech (S2S) models provide natural, low-latency conversational interaction and would benefit from the ability to use external tools and complete voice-agent…
The Internet Archive Music Dataset
Paraskevas Stamatiadis, Bernardo V Miranda, Clémentine Berger +3
We introduce the Internet Archive Music Dataset (IAMD), a large-scale collection of captioned music segments derived from the Internet Archive. To the best of our knowledge, IAMD c…
Unlocking Spatial Grounding in Large Audio-Visual Retrieval models
Hugo Malard, Michel Olvera, Sanjeel Parekh +3
Weak supervision sets a practical regime for audio-visual sound source localization as dense spatial annotations are costly to obtain at scale. The task, however, remains challengi…
S-SONDO: Self-Supervised Knowledge Distillation for General Audio Foundation Models
Mohammed Ali El Adlouni, Aurian Quelennec, Pierre Chouteau +2
General audio foundation models have recently achieved remarkable progress, enabling strong performance across diverse tasks. However, state-of-the-art models remain extremely larg…
TinyMU: A Compact Audio-Language Model for Music Understanding
Xiquan Li, Aurian Quelennec, Slim Essid
Music understanding and reasoning are central challenges in the Music Information Research field, with applications ranging from retrieval and recommendation to music agents and vi…
Contrastive Knowledge Distillation for Embedding Refinement in Personalized Speech Enhancement
Thomas Serre, Mathieu Fontaine, Éric Benhaim +1
Personalized speech enhancement (PSE) has shown convincing results when it comes to extracting a known target voice among interfering ones. The corresponding systems usually incorp…
O-EENC-SD: Efficient Online End-to-End Neural Clustering for Speaker Diarization
Elio Gruttadauria, Mathieu Fontaine, Jonathan Le Roux +1
We introduce O-EENC-SD: an end-to-end online speaker diarization system based on EEND-EDA, featuring a novel RNN-based stitching mechanism for online prediction. In particular, we…
Controlling Contrastive Self-Supervised Learning with Knowledge-Driven Multiple Hypothesis: Application to Beat Tracking
Antonin Gagnere, Slim Essid, Geoffroy Peeters
Ambiguities in data and problem constraints can lead to diverse, equally plausible outcomes for a machine learning task. In beat and downbeat tracking, for instance, different list…
IS : Generic Impulsive--Stationary Sound Separation in Acoustic Scenes using Deep Filtering
Clémentine Berger, Paraskevas Stamatiadis, Roland Badeau +1
We are interested in audio systems capable of performing a differentiated processing of stationary backgrounds and isolated acoustic events within an acoustic scene, whether for ap…
Make me an Expert: Distilling from Generalist Black-Box Models into Specialized Models for Semantic Segmentation
Yasser Benigmim, Subhankar Roy, Khalid Oublal +4
The rise of Artificial Intelligence as a Service (AIaaS) democratizes access to pre-trained models via Application Programming Interfaces (APIs), but also raises a fundamental ques…
MATPAC++: Enhanced Masked Latent Prediction for Self-Supervised Audio Representation Learning
Aurian Quelennec, Pierre Chouteau, Geoffroy Peeters +1
Masked latent prediction has emerged as a leading paradigm in self-supervised learning (SSL), especially for general audio and music representation learning. While recent methods h…
Multiple Choice Learning of Low-Rank Adapters for Language Modeling
Victor Letzelter, Hugo Malard, Mathieu Fontaine +4
We propose LoRA-MCL, a training scheme that extends next-token prediction in language models with a method designed to decode diverse, plausible sentence continuations at inference…
Perceptual Noise-Masking with Music through Deep Spectral Envelope Shaping
Clémentine Berger, Roland Badeau, Slim Essid
People often listen to music in noisy environments, seeking to isolate themselves from ambient sounds. Indeed, a music signal can mask some of the noise's frequency components due…
Masked Latent Prediction and Classification for Self-Supervised Audio Representation Learning
Aurian Quelennec, Pierre Chouteau, Geoffroy Peeters +1
Recently, self-supervised learning methods based on masked latent prediction have proven to encode input data into powerful representations. However, during training, the learned l…
TACO: Training-free Sound Prompted Segmentation via Semantically Constrained Audio-visual CO-factorization
Hugo Malard, Michel Olvera, Stephane Lathuiliere +1
Large-scale pre-trained audio and image models demonstrate an unprecedented degree of generalization, making them suitable for a wide range of applications. Here, we tackle the spe…
Multiple Choice Learning for Efficient Speech Separation with Many Speakers
David Perera, François Derrida, Théo Mariotte +2
Training speech separation models in the supervised setting raises a permutation problem: finding the best assignation between the model predictions and the ground truth separated…
A Contrastive Self-Supervised Learning scheme for beat tracking amenable to few-shot learning
Antonin Gagnere, Geoffroy Peeters, Slim Essid
In this paper, we propose a novel Self-Supervised-Learning scheme to train rhythm analysis systems and instantiate it for few-shot beat tracking. Taking inspiration from the Contra…
An Eye for an Ear: Zero-shot Audio Description Leveraging an Image Captioner using Audiovisual Distribution Alignment
Hugo Malard, Michel Olvera, Stéphane Lathuiliere +1
Multimodal large language models have fueled progress in image captioning. These models, fine-tuned on vast image datasets, exhibit a deep understanding of semantic concepts. In th…
A sound description: Exploring prompt templates and class descriptions to enhance zero-shot audio classification
Michel Olvera, Paraskevas Stamatiadis, Slim Essid
Audio-text models trained via contrastive learning offer a practical approach to perform audio classification through natural language prompts, such as "this is a sound of" followe…
SALT: Standardized Audio event Label Taxonomy
Paraskevas Stamatiadis, Michel Olvera, Slim Essid
Machine listening systems often rely on fixed taxonomies to organize and label audio data, key for training and evaluating deep neural networks (DNNs) and other supervised algorith…
Annealed Multiple Choice Learning: Overcoming limitations of Winner-takes-all with annealing
David Perera, Victor Letzelter, Théo Mariotte +4
We introduce Annealed Multiple Choice Learning (aMCL) which combines simulated annealing with MCL. MCL is a learning framework handling ambiguous tasks by predicting a small set of…
Less Forgetting for Better Generalization: Exploring Continual-learning Fine-tuning Methods for Speech Self-supervised Representations
Salah Zaiem, Titouan Parcollet, Slim Essid
Despite being trained on massive and diverse datasets, speech self-supervised encoders are generally used for downstream purposes as mere frozen feature extractors or model initial…