8 papers
Code-Switching Detection via Language-Selective Dimensions in Speech Foundation Models
Jie Chi, Zakaria Aldeneh, Masha Fedzechkina +2
Code-switching (CS), the alternation between languages within an utterance, is common in multilingual speech but underrepresented in pretraining corpora. Increasing CS coverage req…
Language Discrimination Improves Linguistic Learning in Multilingual Speech Models
Maureen de Seyssel, Jie Chi, Zakaria Aldeneh
Multilingual self-supervised speech models can benefit from sharing information across languages, but under a matched total pretraining data budget they still fall short of monolin…
A Practical Recipe for Semi-Supervised Federated ASR: Online Pseudo-Labels with Server Update Stabilization
Wonho Bae, Zakaria Aldeneh, Martin Pelikan +3
Semi-supervised federated learning (SSFL) trains models on clients' unlabeled data using a teacher to generate pseudo-labels, with a small labeled seed dataset on the server. Autom…
Which Data Matter? Embedding-Based Data Selection for Speech Recognition
Zakaria Aldeneh, Skyler Seto, Maureen de Seyssel +8
Modern ASR systems are typically trained on large-scale pseudo-labeled, in-the-wild data spanning multiple domains. While such heterogeneous data benefit generalist models designed…
Closing the Gap Between Text and Speech Understanding in LLMs
Santiago Cuervo, Skyler Seto, Maureen de Seyssel +5
Large Language Models (LLMs) can be adapted to extend their text capabilities to speech inputs. However, these speech-adapted LLMs consistently underperform their text-based counte…
Leveraging Audio-Visual Data to Reduce the Multilingual Gap in Self-Supervised Speech Models
María Andrea Cruz Blandón, Zakaria Aldeneh, Jie Chi +1
Self-supervised learning (SSL) has made significant advances in speech representation learning. Models like wav2vec 2.0 and HuBERT have achieved state-of-the-art results in tasks s…
Speech Foundation Models Generalize to Time Series Tasks from Wearable Sensor Data
Jaya Narain, Zakaria Aldeneh, Shirley Ren
Both speech and sensor time series data encode information in both the time- and frequency- domains, like spectral powers and waveform shapelets. We show that speech foundation mod…
ChipChat: Low-Latency Cascaded Conversational Agent in MLX
Tatiana Likhomanenko, Richard He Bai, Zijin Gu +8
The emergence of large language models (LLMs) has transformed spoken dialog systems, yet the optimal architecture for real-time on-device voice agents remains an open question. Whi…