479 citations · 1.2k across the 92 of their papers we have counts for
99 papers
BeatGraph: Self-Supervised Heartbeat Graphs for Infant ECG Representations from the Home Environment
Mohammad Nur Hossain Khan, M. S. Krafczyk, Beverly G. Bolster +3
Electrocardiogram (ECG) foundation models typically tokenize the signal into fixed-length patches that ignore cardiac structure, so a patch may split a heartbeat and the number of…
Learning New Words from Unlabeled Test Data in Automatic Speech Recognition
Mengqi Wang, Mark A. Hasegawa-Johnson, Haolong Zheng +1
New words are invented every day. A human listener can learn a new word by hearing it clearly once and inferring its usage from sentence context. This paper proposes granting ASR a…
Unsupervised Speech Recognition at the Syllable Level
Liming Wang, Kai-Wei Chang, Kunio Kashino +3
Training speech recognizers with unpaired speech and text -- known as unsupervised speech recognition (UASR) -- is a crucial step toward extending ASR to low-resource languages in…
Robust Multi-Tier Infant-Centered Audio Understanding with Whisper via Structured Speaker Conditioning
Xulin Fan, Jialu Li, Mohammad Nur Hossain Khan +4
Recent advances in model design and self-supervised audio representations have improved speech and audio understanding, yet infant-centered naturalistic recordings remain challengi…
GigaSpeechBench: A Real-World Multilingual Speech-to-Text Benchmark
Yujie Tu, Yifan Yang, Tianrui Wang +35
While modern ASR systems achieve low error rates on high-resource benchmarks, such performance often overestimates real-world robustness. Existing evaluations address challenges in…
CHILDES-Aligned: A Curated Children's Speech Dataset via Multi-Model Timestamp Ensembling
Haolong Zheng, Yuanzhuo Hu, Xinyu Liang +7
CHILDES is a large-scale child speech corpus containing long-form recordings of naturalistic child-adult interactions, making it a valuable resource for studying child speech and l…
Token-level Response-visual Attention Guidance for Multimodal LLMs Knowledge Distillation
Jaehyun Jang, Eunseop Yoon, Hee Suk Yoon +3
While knowledge distillation (KD) is widely adopted for training lightweight models by leveraging supervision from larger teacher models, relying solely on output token distributio…
SPARCLE: SPeaker-aware Aligned Representations via Contrastive Language Embeddings
Priyam Mazumdar, Yurii Halychanskyi, Steven Guo +2
Recent advances in speech synthesis have shifted from phoneme representations to direct grapheme modeling. While phonemes address the one-to-many mapping between text and acoustics…
Transcript-Free Flow-Matching Text-to-Speech via Speech Feature Conditioning
SooHwan Eom, Hee Suk Yoon, Eunseop Yoon +2
Recent flow-matching text-to-speech (TTS) models, such as F5-TTS, rely on a reference transcript at inference time, obtained from an external ASR system. This dependency makes zero…
Low-Burden Data Augmentation for Dysarthric ASR via Zero-Shot Voice Cloning
Satwinder Singh, Qianli Wang, Zihan Zhong +4
Automatic speech recognition remains unreliable for dysarthric speech due to data scarcity and high inter-speaker variability. While synthetic data can address these gaps, traditio…
SiamCTC: Learning Speech Representations through Monotonic Temporal Alignment
SooHwan Eom, Mark Hasegawa-Johnson, Chang D. Yoo
Self-supervised speech representation learning has made significant progress through Siamese networks, which leverage different views of the same input. However, existing methods o…
Query-based Cross-Modal Projector Bolstering Mamba Multimodal LLM
SooHwan Eom, Jay Shim, Gwanhyeong Koo +4
The Transformer's quadratic complexity with input length imposes an unsustainable computational load on large language models (LLMs). In contrast, the Selective Scan Structured Sta…
FSA-GRPO: Teaching Auditory LLMs to Use Few-Shot Demonstrations
Haolong Zheng, Siyin Wang, Xulin Fan +2
Few-shot prompting provides an effective way to adapt auditory large language models to low-resource tasks such as children's speech recognition. However, most auditory large langu…
Decomposed On-Policy Distillation for Vision-Language Reasoning: Steering Gradients for Visual Grounding
Hee Suk Yoon, Eunseop Yoon, Jaehyun Jang +6
While on-policy distillation offers dense supervision for training small reasoning models, its optimization dynamics in the multimodal domain remain under-explored. In this work, w…
SALSA: Speech Aware LLM Adaptation via Learned Steering Activation Vectors
Yekaterina Yegorova, Argyrios Gerogiannis, Haolong Zheng +3
Speech-aware large language models often generalize poorly to out-of-domain settings. We propose SALSA (Speech-Aware LLM Adaptation via Learned Steering Activations), a lightweight…
LEAF: Growing Trees Without Branching for Speech-Aware Large Language Model Post-Training
Argyrios Gerogiannis, Yekaterina Yegorova, Mark Hasegawa-Johnson +1
State-of-the-art GRPO-style methods for speech-aware large language model post-training suffer from coarse credit assignment, broadcasting the same terminal-reward advantage to eve…
PDCR: Perception-Decomposed Confidence Reward for Vision-Language Reasoning
Hee Suk Yoon, Eunseop Yoon, Ji Woo Hong +6
Reinforcement Learning with Verifiable Rewards (RLVR) traditionally relies on a sparse, outcome-based signal. Recent work shows that providing a fine-grained, model-intrinsic signa…
In-Sync: Adaptation of Speech Aware Large Language Models for ASR with Word Level Timestamp Predictions
Xulin Fan, Vishal Sunder, Samuel Thomas +3
Recent advances in speech-aware language models have coupled strong acoustic encoders with large language models, enabling systems that move beyond transcription to produce richer…
Few-Shot Synthetic Accented Speech for ASR Fine-Tuning: What Helps and When?
Yurii Halychanskyi, Nimet Beyza Bozdag, Mark Hasegawa-Johnson +2
Synthetic accented speech is a promising way to improve automatic speech recognition (ASR) when real accented recordings are scarce. We ask what makes such data useful for ASR fine…
Something from Nothing: Data Augmentation for Robust Severity Level Estimation of Dysarthric Speech
Jaesung Bae, Xiuwen Zheng, Minje Kim +2
Dysarthric speech quality assessment (DSQA) is critical for clinical diagnostics and inclusive speech technologies. However, subjective evaluation is costly and difficult to scale,…
Towards Robust Dysarthric Speech Recognition: LLM-Agent Post-ASR Correction Beyond WER
Xiuwen Zheng, Sixun Dong, Bornali Phukon +2
While Automatic Speech Recognition (ASR) is typically benchmarked by word error rate (WER), real-world applications ultimately hinge on semantic fidelity. This mismatch is particul…
AVMeme Exam: A Multimodal Multilingual Multicultural Benchmark for LLMs' Contextual and Cultural Knowledge and Thinking
Xilin Jiang, Qiaolin Wang, Junkai Wu +30
Internet audio-visual clips convey meaning through time-varying sound and motion, which extend beyond what text alone can represent. To examine whether AI models can understand suc…
MetaSICL: Globalizing Auditory LLMs for Underserved Speakers and Languages via Meta Speech In-Context Learning
Haolong Zheng, Siyin Wang, Zengrui Jin +1
Generative AI for speech and audio is increasingly expected to serve users across languages, cultures, and communities, yet current auditory Large Language Models (LLMs) are still…
TICL+: A Case Study On Speech In-Context Learning for Children's Speech Recognition
Haolong Zheng, Yekaterina Yegorova, Mark Hasegawa-Johnson
Children's speech recognition remains challenging due to substantial acoustic and linguistic variability, limited labeled data, and significant differences from adult speech. Speec…