5 papers
DECAF: Dynamic Envelope Context-Aware Fusion for Speech-Envelope Reconstruction from EEG
Karan Thakkar, Mounya Elhilali
Reconstructing the speech audio envelope from scalp neural recordings (EEG) is a central task for decoding a listener's attentional focus in applications like neuro-steered hearing…
Using LLMs for Late Multimodal Sensor Fusion for Activity Recognition
Ilker Demirel, Karan Thakkar, Benjamin Elizalde +7
Sensor data streams provide valuable information around activities and context for downstream applications, though integrating complementary information can be challenging. We show…
CapSpeech: Enabling Downstream Applications in Style-Captioned Text-to-Speech
Helin Wang, Jiarui Hai, Dading Chong +11
Recent advancements in generative artificial intelligence have significantly transformed the field of style-captioned text-to-speech synthesis (CapTTS). However, adapting CapTTS to…
Foundation Model Hidden Representations for Heart Rate Estimation from Auscultation
Jingping Nie, Dung T. Tran, Karan Thakkar +5
Auscultation, particularly heart sound, is a non-invasive technique that provides essential vital sign information. Recently, self-supervised acoustic representation foundation mod…
SoloAudio: Target Sound Extraction with Language-oriented Audio Diffusion Transformer
Helin Wang, Jiarui Hai, Yen-Ju Lu +3
In this paper, we introduce SoloAudio, a novel diffusion-based generative model for target sound extraction (TSE). Our approach trains latent diffusion models on audio, replacing t…