18 papers
Towards Data-free and Training-free Compression for Speech Foundation Models Using Parameter Clustering
Haoning Xu, Zhaoqing Li, Huimeng Wang +4
This paper presents a novel data-free and training-free compression approach for speech foundation models using channelwise clustering via k-means. More fine-grained, mixed sparsit…
UNISON: A Unified Sound Generation and Editing Framework via Deep LLM Fusion
Zhaoqing Li, Haoning Xu, Jingran Su +9
We present UNISON, a latent diffusion framework that unifies speech generation, sound generation, and audio editing within a single model. A single model handles text-to-audio, tex…
SemaVoice: Semantic-Aware Continuous Autoregressive Speech Synthesis
Huimeng Wang, Hui Lu, Jiajun Deng +7
Continuous autoregressive speech synthesis has recently emerged as a promising direction for zero-shot text-to-speech (TTS). However, existing methods still suffer from a fundament…
Multi-Channel Speech Enhancement for Cocktail Party Speech Emotion Recognition
Youjun Chen, Guinan Li, Mengzhe Geng +9
This paper highlights the critical importance of multi-channel speech enhancement (MCSE) for speech emotion recognition (ER) in cocktail party scenarios. A multi-channel speech der…
Towards Effective and Efficient Non-autoregressive decoders for Conformer and LLM-based ASR using Block-based Attention Mask
Tianzi Wang, Xurong Xie, Zengrui Jin +9
Automatic speech recognition (ASR) systems often rely on autoregressive (AR) Transformer decoder architectures, which limit efficient inference parallelization due to their sequent…
Exploring Cross-Utterance Speech Contexts for Conformer-Transducer Speech Recognition Systems
Mingyu Cui, Mengzhe Geng, Jiajun Deng +8
This paper investigates four types of cross-utterance speech contexts modeling approaches for streaming and non-streaming Conformer-Transformer (C-T) ASR systems: i) input audio fe…
Effective and Efficient One-pass Compression of Speech Foundation Models Using Sparsity-aware Self-pinching Gates
Haoning Xu, Zhaoqing Li, Youjun Chen +5
This paper presents a novel approach for speech foundation models compression that tightly integrates model pruning and parameter update into a single stage. Highly compact layer-l…
Towards One-bit ASR: Extremely Low-bit Conformer Quantization Using Co-training and Stochastic Precision
Zhaoqing Li, Haoning Xu, Zengrui Jin +7
Model compression has become an emerging need as the sizes of modern speech systems rapidly increase. In this paper, we study model weight quantization, which directly reduces the…
Unfolding A Few Structures for The Many: Memory-Efficient Compression of Conformer and Speech Foundation Models
Zhaoqing Li, Haoning Xu, Xurong Xie +3
This paper presents a novel memory-efficient model compression approach for Conformer ASR and speech foundation systems. Our approach features a unique "small-to-large" design. A c…
Effective and Efficient Mixed Precision Quantization of Speech Foundation Models
Haoning Xu, Zhaoqing Li, Zengrui Jin +7
This paper presents a novel mixed-precision quantization approach for speech foundation models that tightly integrates mixed-precision learning and quantized model parameter estima…
Phone-purity Guided Discrete Tokens for Dysarthric Speech Recognition
Huimeng Wang, Xurong Xie, Mengzhe Geng +6
Discrete tokens extracted provide efficient and domain adaptable speech features. Their application to disordered speech that exhibits articulation imprecision and large mismatch a…
Structured Speaker-Deficiency Adaptation of Foundation Models for Dysarthric and Elderly Speech Recognition
Shujie Hu, Xurong Xie, Mengzhe Geng +8
Data-intensive fine-tuning of speech foundation models (SFMs) to scarce and diverse dysarthric and elderly speech leads to data bias and poor generalization to unseen speakers. Thi…
CR-CTC: Consistency regularization on CTC for improved speech recognition
Zengwei Yao, Wei Kang, Xiaoyu Yang +7
Connectionist Temporal Classification (CTC) is a widely used method for automatic speech recognition (ASR), renowned for its simplicity and computational efficiency. However, it of…
Exploring SSL Discrete Speech Features for Zipformer-based Contextual ASR
Mingyu Cui, Yifan Yang, Jiajun Deng +7
Self-supervised learning (SSL) based discrete speech representations are highly compact and domain adaptable. In this paper, SSL discrete speech features extracted from WavLM model…
Large Language Model Can Transcribe Speech in Multi-Talker Scenarios with Versatile Instructions
Lingwei Meng, Shujie Hu, Jiawen Kang +6
Recent advancements in large language models (LLMs) have revolutionized various domains, bringing significant progress and new opportunities. Despite progress in speech-related tas…
Homogeneous Speaker Features for On-the-Fly Dysarthric and Elderly Speaker Adaptation
Mengzhe Geng, Xurong Xie, Jiajun Deng +7
The application of data-intensive automatic speech recognition (ASR) technologies to dysarthric and elderly adult speech is confronted by their mismatch against healthy and nonaged…
One-pass Multiple Conformer and Foundation Speech Systems Compression and Quantization Using An All-in-one Neural Model
Zhaoqing Li, Haoning Xu, Tianzi Wang +7
We propose a novel one-pass multiple ASR systems joint compression and quantization approach using an all-in-one neural model. A single compression cycle allows multiple nested sys…
Towards Effective and Efficient Non-autoregressive Decoding Using Block-based Attention Mask
Tianzi Wang, Xurong Xie, Zhaoqing Li +9
This paper proposes a novel non-autoregressive (NAR) block-based Attention Mask Decoder (AMD) that flexibly balances performance-efficiency trade-offs for Conformer ASR systems. AM…