38 citations · 88 across the 46 of their papers we have counts for
47 papers
Augmenting Dysarthric Speech Severity Assessment with MOS Supervision
Kaimeng Jia, Minzhu Tu, Zengrui Jin +2
Dysarthria is a speech disorder marked by reduced intelligibility and communicative effectiveness. Automatic utterance-level assessment of dysarthric speech can support scalable sp…
Explainable and Trustworthy Speech Emotion Recognition Using Confidence Score and Reinforcement Learning Rectified Speech Emotion Descriptors
Youjun Chen, Xurong Xie, Mengzhe Geng +9
Explainable and trustworthy speech emotion recognition (SER) remains a challenging task to date, largely due to the scarcity of SER data with reliable speech emotion descriptor (SE…
UNISON: A Unified Sound Generation and Editing Framework via Deep LLM Fusion
Zhaoqing Li, Haoning Xu, Jingran Su +9
We present UNISON, a latent diffusion framework that unifies speech generation, sound generation, and audio editing within a single model. A single model handles text-to-audio, tex…
FSA-GRPO: Teaching Auditory LLMs to Use Few-Shot Demonstrations
Haolong Zheng, Siyin Wang, Xulin Fan +2
Few-shot prompting provides an effective way to adapt auditory large language models to low-resource tasks such as children's speech recognition. However, most auditory large langu…
Optimising Neural Speech Codecs for 300bps Communication using Reinforcement Learning
Junyi Wang, Chi Zhang, Jing Qian +4
In bandwidth-constrained communication such as satellite and underwater channels, speech must often be transmitted at ultra-low bitrates where intelligibility is the primary object…
ClariCodec: Optimising Neural Speech Codes for 200bps Communication using Reinforcement Learning
Junyi Wang, Chi Zhang, Jing Qian +4
In bandwidth-constrained communication such as satellite and underwater channels, speech must often be transmitted at ultra-low bitrates where intelligibility is the primary object…
sleep2vec: Unified Cross-Modal Alignment for Heterogeneous Nocturnal Biosignals
Weixuan Yuan, Zengrui Jin, Yichen Wang +4
Tasks ranging from sleep staging to clinical diagnosis traditionally rely on standard polysomnography (PSG) devices, bedside monitors and wearable devices, which capture diverse no…
JAEGER: Joint 3D Audio-Visual Grounding and Reasoning in Simulated Physical Environments
Zhan Liu, Changli Tang, Yuxin Wang +7
Current audio-visual large language models (AV-LLMs) are predominantly restricted to 2D perception, relying on RGB video and monaural audio. This design choice introduces a fundame…
Discovery and Reinforcement of Tool-Integrated Reasoning Chains via Rollout Trees
Kun Li, Zenan Xu, Junan Li +4
Tool-Integrated Reasoning has emerged as a key paradigm to augment Large Language Models (LLMs) with computational capabilities, yet integrating tool-use into long Chain-of-Thought…
MetaSICL: Globalizing Auditory LLMs for Underserved Speakers and Languages via Meta Speech In-Context Learning
Haolong Zheng, Siyin Wang, Zengrui Jin +1
Generative AI for speech and audio is increasingly expected to serve users across languages, cultures, and communities, yet current auditory Large Language Models (LLMs) are still…
Towards Fine-Grained and Multi-Granular Contrastive Language-Speech Pre-training
Yifan Yang, Bing Han, Hui Wang +8
Modeling fine-grained speaking styles remains challenging for language-speech representation pre-training, as existing speech-text models are typically trained with coarse captions…
Towards Effective and Efficient Non-autoregressive decoders for Conformer and LLM-based ASR using Block-based Attention Mask
Tianzi Wang, Xurong Xie, Zengrui Jin +9
Automatic speech recognition (ASR) systems often rely on autoregressive (AR) Transformer decoder architectures, which limit efficient inference parallelization due to their sequent…
Augmenting Open-Vocabulary Dysarthric Speech Assessment with Human Perceptual Supervision
Kaimeng Jia, Minzhu Tu, Zengrui Jin +2
Dysarthria is a speech disorder characterized by impaired intelligibility and reduced communicative effectiveness. Automatic dysarthria assessment provides a scalable, cost-effecti…
Towards General Auditory Intelligence: Large Multimodal Models for Machine Listening and Speaking
Siyin Wang, Zengrui Jin, Changli Tang +26
In the era of large language models (LLMs) and artificial general intelligence (AGI), computer audition must evolve beyond traditional paradigms to fully leverage the capabilities…
Speech-Audio Compositional Attacks on Multimodal LLMs and Their Mitigation with SALMONN-Guard
Yudong Yang, Xuezhen Zhang, Zhifeng Han +6
Recent progress in LLMs has enabled understanding of audio signals, but has also exposed new safety risks arising from complex audio inputs that are inadequately handled by current…
Multiplexing Neural Audio Watermarks with Adaptive Routing
Zheqi Yuan, Yucheng Huang, Guangzhi Sun +2
Audio watermarking supports speech authenticity verification. We study whether combining heterogeneous watermarks can retain at least one detectable provenance signal when their fa…
SPEAR: A Unified SSL Framework for Learning Speech and Audio Representations
Xiaoyu Yang, Yifan Yang, Zengrui Jin +5
Self-supervised learning (SSL) has significantly advanced acoustic representation learning. However, most existing models are optimised for either speech or audio event understandi…
Audio-Conditioned Diffusion LLMs for ASR and Deliberation Processing
Mengqi Wang, Zhan Liu, Zengrui Jin +3
Diffusion-based large language models (DLLMs) have recently attracted growing interest as an alternative to autoregressive decoders. In this work, we present an empirical study on…
Towards One-bit ASR: Extremely Low-bit Conformer Quantization Using Co-training and Stochastic Precision
Zhaoqing Li, Haoning Xu, Zengrui Jin +7
Model compression has become an emerging need as the sizes of modern speech systems rapidly increase. In this paper, we study model weight quantization, which directly reduces the…
Unfolding A Few Structures for The Many: Memory-Efficient Compression of Conformer and Speech Foundation Models
Zhaoqing Li, Haoning Xu, Xurong Xie +3
This paper presents a novel memory-efficient model compression approach for Conformer ASR and speech foundation systems. Our approach features a unique "small-to-large" design. A c…
Effective and Efficient Mixed Precision Quantization of Speech Foundation Models
Haoning Xu, Zhaoqing Li, Zengrui Jin +7
This paper presents a novel mixed-precision quantization approach for speech foundation models that tightly integrates mixed-precision learning and quantized model parameter estima…
Structured Speaker-Deficiency Adaptation of Foundation Models for Dysarthric and Elderly Speech Recognition
Shujie Hu, Xurong Xie, Mengzhe Geng +8
Data-intensive fine-tuning of speech foundation models (SFMs) to scarce and diverse dysarthric and elderly speech leads to data bias and poor generalization to unseen speakers. Thi…
k2SSL: A Faster and Better Framework for Self-Supervised Speech Representation Learning
Yifan Yang, Jianheng Zhuo, Zengrui Jin +9
Self-supervised learning (SSL) has achieved great success in speech-related tasks. While Transformer and Conformer architectures have dominated SSL backbones, encoders like Zipform…
CR-CTC: Consistency regularization on CTC for improved speech recognition
Zengwei Yao, Wei Kang, Xiaoyu Yang +7
Connectionist Temporal Classification (CTC) is a widely used method for automatic speech recognition (ASR), renowned for its simplicity and computational efficiency. However, it of…