45 citations · 211 across the 23 of their papers we have counts for
27 papers
SepRQ : Self-Supervised Speech Mixture Representation Learning via Mask-Free, Multi-Scale Source Separation
Séverin Baroudi, Hervé Bredin, Ricard Marxer
Self-supervised learning (SSL) is standard for speech representation learning, but mainstream models are designed around single-speaker audio, limiting their usefulness in multi-sp…
Steering Speech-Language Models: Training-Free Task Specialization via Contrastive Activation Addition
Séverin Baroudi, Yanis Labrak, Pierfrancesco Melucci +5
Activation steering has proven effective for controlling the behavior of Large Language Models (LLMs) at inference time, but its application to SpeechLLMs remains new, and training…
GLaS-JEPA: Gaussian-Regularized Speech SSL without Engineered Prediction Targets
Gaspard Botté, Séverin Baroudi, Samir Sadok +5
Speech self-supervised learning aims to learn general-purpose representations for downstream speech tasks. However, current approaches rely on complex, carefully designed predictio…
Generating Synthetic Doctor-Patient Conversations for Long-form Audio Summarization
Yanis Labrak, David Grünert, Séverin Baroudi +11
Long-context audio reasoning is underserved in both training data and evaluation. Existing benchmarks target short-context tasks, and the open-ended generation tasks most relevant…
DASHA: Diarization-Aware Speech-LLM ASR for Healthcare in Code-Switched Indic Doctor-Patient Conversations
Séverin Baroudi, Yanis Labrak, Shashi Kumar +7
Extracting patients' medical conditions from recorded primary-care conversations requires knowing who said what, which is difficult when speech is far-field, spontaneous, overlappi…
On the Use of Self-Supervised Representation Learning for Speaker Diarization and Separation
Séverin Baroudi, Hervé Bredin, Joseph Razik +1
Self-supervised speech models such as wav2vec2.0 and WavLM have been shown to significantly improve the performance of many downstream speech tasks, especially in low-resource sett…
Factorized RVQ-GAN For Disentangled Speech Tokenization
Sameer Khurana, Dominik Klement, Antoine Laurent +13
We propose Hierarchical Audio Codec (HAC), a unified neural speech codec that factorizes its bottleneck into three linguistic levels-acoustic, phonetic, and lexical-within a single…
Aligning Multimodal Representations through an Information Bottleneck
Antonio Almudévar, José Miguel Hernández-Lobato, Sameer Khurana +2
Contrastive losses have been extensively used as a tool for multimodal representation learning. However, it has been empirically observed that their use is not effective to learn a…
Discrete Audio Tokens: More Than a Survey!
Pooneh Mousavi, Gallil Maimon, Adel Moumen +18
Discrete audio tokens are compact representations that aim to preserve perceptual quality, phonetic content, and speaker characteristics while enabling efficient storage and infere…
LFAR: Accounting for Layerwise Dynamics to Improve Multimodal Adaptation of Language Models
Santiago Cuervo, Adel Moumen, Yanis Labrak +5
Text-pretrained language models (LMs) encode rich world knowledge, but adapting them to process and generate perceptual modalities such as audio and images while effectively levera…
TalTech-IRIT-LIS Speaker and Language Diarization Systems for DISPLACE 2024
Joonas Kalda, Tanel Alumäe, Martin Lebourdais +3
This paper describes the submissions of team TalTech-IRIT-LIS to the DISPLACE 2024 challenge. Our team participated in the speaker diarization and language diarization tracks of th…
Scaling Properties of Speech Language Models
Santiago Cuervo, Ricard Marxer
Speech Language Models (SLMs) aim to learn language from raw audio, without textual resources. Despite significant advances, our current models exhibit weak syntax and semantic abi…
Transfer Learning from Whisper for Microscopic Intelligibility Prediction
Paul Best, Santiago Cuervo, Ricard Marxer
Macroscopic intelligibility models predict the expected human word-error-rate for a given speech-in-noise stimulus. In contrast, microscopic intelligibility models aim to make fine…
PixIT: Joint Training of Speaker Diarization and Speech Separation from Real-world Multi-speaker Recordings
Joonas Kalda, Clément Pagés, Ricard Marxer +2
A major drawback of supervised speech separation (SSep) systems is their reliance on synthetic data, leading to poor real-world generalization. Mixture invariant training (MixIT) w…
Speech foundation models on intelligibility prediction for hearing-impaired listeners
Santiago Cuervo, Ricard Marxer
Speech foundation models (SFMs) have been benchmarked on many speech processing tasks, often achieving state-of-the-art performance with minimal adaptation. However, the SFM paradi…
Eiffel Tower: A Deep-Sea Underwater Dataset for Long-Term Visual Localization
Clémentin Boittiaux, Claire Dune, Maxime Ferrera +5
Visual localization plays an important role in the positioning and navigation of robotics systems within previously visited environments. When visits occur over long periods of tim…
SUCRe: Leveraging Scene Structure for Underwater Color Restoration
Clémentin Boittiaux, Ricard Marxer, Claire Dune +3
Underwater images are altered by the physical characteristics of the medium through which light rays pass before reaching the optical sensor. Scattering and wavelength-dependent ab…
Variable-rate hierarchical CPC leads to acoustic unit discovery in speech
Santiago Cuervo, Adrian Łańcucki, Ricard Marxer +2
The success of deep learning comes from its ability to capture the hierarchical structure of data by learning high-level representations defined in terms of low-level ones. In this…
Homography-Based Loss Function for Camera Pose Regression
Clémentin Boittiaux, Ricard Marxer, Claire Dune +2
Some recent visual-based relocalization algorithms rely on deep learning methods to perform camera pose regression from image data. This paper focuses on the loss functions that em…
Contrastive prediction strategies for unsupervised segmentation and categorization of phonemes and words
Santiago Cuervo, Maciej Grabias, Jan Chorowski +4
We investigate the performance on phoneme categorization and phoneme and word segmentation of several self-supervised learning (SSL) methods based on Contrastive Predictive Coding…
Information Retrieval for ZeroSpeech 2021: The Submission by University of Wroclaw
Jan Chorowski, Grzegorz Ciesielski, Jarosław Dzikowski +6
We present a number of low-resource approaches to the tasks of the Zero Resource Speech Challenge 2021. We build on the unsupervised representations of speech proposed by the organ…
Aligned Contrastive Predictive Coding
Jan Chorowski, Grzegorz Ciesielski, Jarosław Dzikowski +6
We investigate the possibility of forcing a self-supervised model trained using a contrastive predictive loss to extract slowly varying latent representations. Rather than producin…
A Convolutional Deep Markov Model for Unsupervised Speech Representation Learning
Sameer Khurana, Antoine Laurent, Wei-Ning Hsu +4
Probabilistic Latent Variable Models (LVMs) provide an alternative to self-supervised learning approaches for linguistic representation learning from speech. LVMs admit an intuitiv…
Robust Training of Vector Quantized Bottleneck Models
Adrian Łańcucki, Jan Chorowski, Guillaume Sanchez +6
In this paper we demonstrate methods for reliable and efficient training of discrete representation using Vector-Quantized Variational Auto-Encoder models (VQ-VAEs). Discrete laten…