6 citations · 18 across the 22 of their papers we have counts for
25 papers
Step-Audio-R1 Technical Report
Fei Tian, Xiangyu Tony Zhang, Yuxin Zhang +14
Recent advances in reasoning models have demonstrated remarkable success in text and vision domains through extended chain-of-thought deliberation. However, a perplexing phenomenon…
Step-Audio-EditX Technical Report
Chao Yan, Boyong Wu, Peng Yang +12
We present Step-Audio-EditX, the first open-source LLM-based audio model excelling at expressive and iterative audio editing encompassing emotion, speaking style, and paralinguisti…
MULTI-Bench: A Multi-Turn Interactive Benchmark for Assessing Emotional Intelligence ability of Spoken Dialogue Models
Yayue Deng, Guoqiang Hu, Haiyang Sun +6
Spoken Dialogue Models (SDMs) have advanced rapidly, yet their ability to sustain genuinely interactive multi-turn conversations remains underexplored, as most benchmarks focus on…
Mind-Paced Speaking: A Dual-Brain Approach to Real-Time Reasoning in Spoken Language Models
Donghang Wu, Haoyang Zhang, Jun Chen +9
Real-time Spoken Language Models (SLMs) struggle to leverage Chain-of-Thought (CoT) reasoning due to the prohibitive latency of generating the entire thought process sequentially.…
Beyond Video-to-SFX: Video to Audio Synthesis with Environmentally Aware Speech
Xinlei Niu, Jianbo Ma, Dylan Harper-Harris +3
The generation of realistic, context-aware audio is important in real-world applications such as video game development. While existing video-to-audio (V2A) methods mainly focus on…
Code-switching Speech Recognition Under the Lens: Model- and Data-Centric Perspectives
Hexin Liu, Haoyang Zhang, Qiquan Zhang +4
Code-switching automatic speech recognition (CS-ASR) presents unique challenges due to language confusion introduced by spontaneous intra-sentence switching and accent bias that bl…
Step-Audio 2 Technical Report
Boyong Wu, Chao Yan, Chen Hu +106
This paper presents Step-Audio 2, an end-to-end multi-modal large language model designed for industry-strength audio understanding and speech conversation. By integrating a latent…
Long-Context Modeling Networks for Monaural Speech Enhancement: A Comparative Study
Qiquan Zhang, Moran Chen, Zeyang Song +3
Advanced long-context modeling backbone networks, such as Transformer, Conformer, and Mamba, have demonstrated state-of-the-art performance in speech enhancement. However, a system…
Why Pre-trained Models Fail: Feature Entanglement in Multi-modal Depression Detection
Xiangyu Zhang, Beena Ahmed, Julien Epps
Depression remains a pressing global mental health issue, driving considerable research into AI-driven detection approaches. While pre-trained models, particularly speech self-supe…
SpeechT-RAG: Reliable Depression Detection in LLMs with Retrieval-Augmented Generation Using Speech Timing Information
Xiangyu Zhang, Hexin Liu, Qiquan Zhang +2
Large Language Models (LLMs) have been increasingly adopted for health-related tasks, yet their performance in depression detection remains limited when relying solely on text inpu…
Selective State Space Model for Monaural Speech Enhancement
Moran Chen, Qiquan Zhang, Mingjiang Wang +4
Voice user interfaces (VUIs) have facilitated the efficient interactions between humans and machines through spoken commands. Since real-word acoustic scenes are complex, speech en…
Rethinking Mamba in Speech Processing by Self-Supervised Models
Xiangyu Zhang, Jianbo Ma, Mostafa Shahin +2
The Mamba-based model has demonstrated outstanding performance across tasks in computer vision, natural language processing, and speech processing. However, in the realm of speech…
Auto-Landmark: Acoustic Landmark Dataset and Open-Source Toolkit for Landmark Extraction
Xiangyu Zhang, Daijiao Liu, Tianyi Xiao +5
In the speech signal, acoustic landmarks identify times when the acoustic manifestations of the linguistically motivated distinctive features are most salient. Acoustic landmarks h…
Binaural Selective Attention Model for Target Speaker Extraction
Hanyu Meng, Qiquan Zhang, Xiangyu Zhang +2
The remarkable ability of humans to selectively focus on a target speaker in cocktail party scenarios is facilitated by binaural audio processing. In this paper, we present a binau…
Mamba in Speech: Towards an Alternative to Self-Attention
Xiangyu Zhang, Qiquan Zhang, Hexin Liu +6
Transformer and its derivatives have achieved success in diverse tasks across computer vision, natural language processing, and speech processing. To reduce the complexity of compu…
Aligning Speech to Languages to Enhance Code-switching Speech Recognition
Hexin Liu, Xiangyu Zhang, Haoyang Zhang +4
Code-switching (CS) refers to the switching of languages within a speech signal and results in language confusion for automatic speech recognition (ASR). To address language confus…
When LLMs Meets Acoustic Landmarks: An Efficient Approach to Integrate Speech into Large Language Models for Depression Detection
Xiangyu Zhang, Hexin Liu, Kaishuai Xu +4
Depression is a critical concern in global mental health, prompting extensive research into AI-based detection methods. Among various AI technologies, Large Language Models (LLMs)…
Speaking in Wavelet Domain: A Simple and Efficient Approach to Speed up Speech Diffusion Model
Xiangyu Zhang, Daijiao Liu, Hexin Liu +5
Recently, Denoising Diffusion Probabilistic Models (DDPMs) have attained leading performances across a diverse range of generative tasks. However, in the field of speech synthesis,…
A Quantitative Approach to Understand Self-Supervised Models as Cross-lingual Feature Extractors
Shuyue Stella Li, Beining Xu, Xiangyu Zhang +3
In this work, we study the features extracted by English self-supervised learning (SSL) models in cross-lingual contexts and propose a new metric to predict the quality of feature…
Enhancing Code-switching Speech Recognition with Interactive Language Biases
Hexin Liu, Leibny Paola Garcia, Xiangyu Zhang +2
Languages usually switch within a multilingual speech signal, especially in a bilingual society. This phenomenon is referred to as code-switching (CS), making automatic speech reco…
Unidirectional brain-computer interface: Artificial neural network encoding natural images to fMRI response in the visual cortex
Ruixing Liang, Xiangyu Zhang, Qiong Li +7
While significant advancements in artificial intelligence (AI) have catalyzed progress across various domains, its full potential in understanding visual perception remains underex…
MERLIon CCS Challenge: A English-Mandarin code-switching child-directed speech corpus for language identification and diarization
Victoria Y. H. Chua, Hexin Liu, Leibny Paola Garcia Perera +7
To enhance the reliability and robustness of language identification (LID) and language diarization (LD) systems for heterogeneous populations and scenarios, there is a need for sp…
Twin-S: A Digital Twin for Skull-base Surgery
Hongchao Shu, Ruixing Liang, Zhaoshuo Li +9
Purpose: Digital twins are virtual interactive models of the real world, exhibiting identical behavior and properties. In surgical applications, computational analysis from digital…
A New Approach to Extract Fetal Electrocardiogram Using Affine Combination of Adaptive Filters
Yu Xuan, Xiangyu Zhang, Shuyue Stella Li +4
The detection of abnormal fetal heartbeats during pregnancy is important for monitoring the health conditions of the fetus. While adult ECG has made several advances in modern medi…