6 citations · 14 across the 30 of their papers we have counts for
31 papers
Agentic-DuplexGen: Decoupling Content, Timing, and Acoustics for Synthetic Dialogue Speech
Pengcheng Wang, Sheng Li, Jiyi Li +1
Synthetic conversational speech has become an important resource for developing and evaluating conversational speech systems. However, existing dialogue synthesis pipelines typical…
Cached LLM Probability Retrieval for Speech Recognition
Sheng Li, Takahiro Shinozaki, Tatsuya Kawahara
Large language models (LLMs) enhance automatic speech recognition (ASR) by providing linguistic priors; however, their direct rescoring is costly because it requires evaluating eve…
Casting Everything to Online API Services? A Survey of Integrating Localized Speech Recognition Models in Robotic Systems
Sheng Li, Jing Li, Felix Schijve +2
Automatic speech recognition (ASR) has become a critical component of modern robotic systems because it is one of the most natural and intuitive ways for humans to interact with ro…
Synchronized Three-Dimensional Vocal-Tract Motion for Speech Synchronization via Joint-Embedding Predictive Architecture Alignment
Sheng Li, Takahiro Shinozaki
Modern neural speech systems can generate intelligible waveforms, but they usually hide the physical speech-production state that produced the sound. Conversely, biomechanical voca…
Emotional Text-To-Speech Based on Mutual-Information-Guided Emotion-Timbre Disentanglement
Jianing Yang, Sheng Li, Takahiro Shinozaki +2
Current emotional Text-To-Speech (TTS) and style transfer methods rely on reference encoders to control global style or emotion vectors, but do not capture nuanced acoustic details…
RAG-Boost: Retrieval-Augmented Generation Enhanced LLM-based Speech Recognition
Pengcheng Wang, Sheng Li, Takahiro Shinozaki
In this paper, we propose RAG-Boost (ST-ShinozakiLab Task I system), which enhances the baseline LLM-based ASR system of the MLC-SLM Challenge (task I) with a retrieval-augmented g…
Towards Emotion Co-regulation with LLM-powered Socially Assistive Robots: Integrating LLM Prompts and Robotic Behaviors to Support Parent-Neurodivergent Child Dyads
Jing Li, Felix Schijve, Sheng Li +3
Socially Assistive Robotics (SAR) has shown promise in supporting emotion regulation for neurodivergent children. Recently, there has been increasing interest in leveraging advance…
End-to-end Acoustic-linguistic Emotion and Intent Recognition Enhanced by Semi-supervised Learning
Zhao Ren, Rathi Adarshi Rammohan, Kevin Scheck +2
Emotion and intent recognition from speech is essential and has been widely investigated in human-computer interaction. The rapid development of social media platforms, chatbots, a…
SpeechIQ: Speech-Agentic Intelligence Quotient Across Cognitive Levels in Voice Understanding by Large Language Models
Zhen Wan, Chao-Han Huck Yang, Yahan Yu +8
We introduce Speech-based Intelligence Quotient (SIQ) as a new form of human cognition-inspired evaluation pipeline for voice understanding large language models, LLM Voice, design…
A Unified Speech LLM for Diarization and Speech Recognition in Multilingual Conversations
Phurich Saengthong, Boonnithi Jiaramaneepinit, Sheng Li +2
Speech Large Language Models (Speech LLMs) have emerged as a crucial paradigm in recent years, extending the capabilities of traditional LLMs to speech tasks such as automatic spee…
Language-Aware Prompt Tuning for Parameter-Efficient Seamless Language Expansion in Multilingual ASR
Hongli Yang, Sheng Li, Hao Huang +2
Recent advancements in multilingual automatic speech recognition (ASR) have been driven by large-scale end-to-end models like Whisper. However, challenges such as language interfer…
Adapting Whisper for Parameter-efficient Code-Switching Speech Recognition via Soft Prompt Tuning
Hongli Yang, Yizhou Peng, Hao Huang +1
Large-scale multilingual ASR models like Whisper excel in high-resource settings but face challenges in low-resource scenarios, such as rare languages and code-switching (CS), due…
Generalized Multilingual Text-to-Speech Generation with Language-Aware Style Adaptation
Haowei Lou, Hye-young Paik, Sheng Li +2
Text-to-Speech (TTS) models can generate natural, human-like speech across multiple languages by transforming phonemes into waveforms. However, multilingual TTS remains challenging…
Joint Automatic Speech Recognition And Structure Learning For Better Speech Understanding
Jiliang Hu, Zuchao Li, Mengjia Shen +3
Spoken language understanding (SLU) is a structure prediction task in the field of speech. Recently, many works on SLU that treat it as a sequence-to-sequence task have achieved gr…
Cross-lingual Embedding Clustering for Hierarchical Softmax in Low-Resource Multilingual Speech Recognition
Zhengdong Yang, Qianying Liu, Sheng Li +2
We present a novel approach centered on the decoding stage of Automatic Speech Recognition (ASR) that enhances multilingual performance, especially for low-resource languages. It u…
Extracting Spatiotemporal Data from Gradients with Large Language Models
Lele Zheng, Yang Cao, Renhe Jiang +4
Recent works show that sensitive user data can be reconstructed from gradient updates, breaking the key privacy promise of federated learning. While success was demonstrated primar…
Investigating Effective Speaker Property Privacy Protection in Federated Learning for Speech Emotion Recognition
Chao Tan, Sheng Li, Yang Cao +2
Federated Learning (FL) is a privacy-preserving approach that allows servers to aggregate distributed models transmitted from local clients rather than training on user data. More…
Benchmarking Japanese Speech Recognition on ASR-LLM Setups with Multi-Pass Augmented Generative Error Correction
Yuka Ko, Sheng Li, Chao-Han Huck Yang +1
With the strong representational power of large language models (LLMs), generative error correction (GER) for automatic speech recognition (ASR) aims to provide semantic and phonet…
Enhancing Privacy of Spatiotemporal Federated Learning against Gradient Inversion Attacks
Lele Zheng, Yang Cao, Renhe Jiang +4
Spatiotemporal federated learning has recently raised intensive studies due to its ability to train valuable models with only shared gradients in various location-based services. O…
MOS-FAD: Improving Fake Audio Detection Via Automatic Mean Opinion Score Prediction
Wangjin Zhou, Zhengdong Yang, Chenhui Chu +4
Automatic Mean Opinion Score (MOS) prediction is employed to evaluate the quality of synthetic speech. This study extends the application of predicted MOS to the task of Fake Audio…
FedCPC: An Effective Federated Contrastive Learning Method for Privacy Preserving Early-Stage Alzheimer's Speech Detection
Wenqing Wei, Zhengdong Yang, Yuan Gao +4
The early-stage Alzheimer's disease (AD) detection has been considered an important field of medical studies. Like traditional machine learning methods, speech-based automatic dete…
GhostVec: A New Threat to Speaker Privacy of End-to-End Speech Recognition System
Xiaojiao Chen, Sheng Li, Jiyi Li +3
Speaker adaptation systems face privacy concerns, for such systems are trained on private datasets and often overfitting. This paper demonstrates that an attacker can extract speak…
Reprogramming Self-supervised Learning-based Speech Representations for Speaker Anonymization
Xiaojiao Chen, Sheng Li, Jiyi Li +3
Current speaker anonymization methods, especially with self-supervised learning (SSL) models, require massive computational resources when hiding speaker identity. This paper propo…
LE-SSL-MOS: Self-Supervised Learning MOS Prediction with Listener Enhancement
Zili Qi, Xinhui Hu, Wangjin Zhou +4
Recently, researchers have shown an increasing interest in automatically predicting the subjective evaluation for speech synthesis systems. This prediction is a challenging task, e…