20 citations · 100 across the 52 of their papers we have counts for
15 papers · 1 filter
SALMONN-2: Advancing General-Purpose Hearing Abilities with Self-Supervised Representations
Xiaoyu Yang, Xuenan Xu, Wenyi Yu +10
Recent audio large language models (ALLMs) are typically built upon audio encoders trained with large amounts of supervised data. Since self-supervised learning (SSL) audio encoder…
Towards General Auditory Intelligence: Large Multimodal Models for Machine Listening and Speaking
Siyin Wang, Zengrui Jin, Changli Tang +26
In the era of large language models (LLMs) and artificial general intelligence (AGI), computer audition must evolve beyond traditional paradigms to fully leverage the capabilities…
Multiplexing Neural Audio Watermarks with Adaptive Routing
Zheqi Yuan, Yucheng Huang, Guangzhi Sun +2
Audio watermarking supports speech authenticity verification. We study whether combining heterogeneous watermarks can retain at least one detectable provenance signal when their fa…
Audio-Conditioned Diffusion LLMs for ASR and Deliberation Processing
Mengqi Wang, Zhan Liu, Zengrui Jin +3
Diffusion-based large language models (DLLMs) have recently attracted growing interest as an alternative to autoregressive decoders. In this work, we present an empirical study on…
SALMONN-omni: A Codec-free LLM for Full-duplex Speech Understanding and Generation
Wenyi Yu, Siyin Wang, Xiaoyu Yang +7
Full-duplex multimodal large language models (LLMs) provide a unified framework for addressing diverse speech understanding and generation tasks, enabling more natural and seamless…
Enabling Auditory Large Language Models for Automatic Speech Quality Evaluation
Siyin Wang, Wenyi Yu, Yudong Yang +10
Speech quality assessment typically requires evaluating audio from multiple aspects, such as mean opinion score (MOS) and speaker similarity (SIM) \etc., which can be challenging t…
Extract and Diffuse: Latent Integration for Improved Diffusion-based Speech and Vocal Enhancement
Yudong Yang, Zhan Liu, Wenyi Yu +3
Diffusion-based generative models have recently achieved remarkable results in speech and vocal enhancement due to their ability to model complex speech data distributions. While t…
SOT Triggered Neural Clustering for Speaker Attributed ASR
Xianrui Zheng, Guangzhi Sun, Chao Zhang +1
This paper introduces a novel approach to speaker-attributed ASR transcription using a neural clustering method. With a parallel processing mechanism, diarisation and ASR can be ap…
Wav2Prompt: End-to-End Speech Prompt Generation and Tuning For LLM in Zero and Few-shot Learning
Keqi Deng, Guangzhi Sun, Philip C. Woodland
Wav2Prompt is proposed which allows straightforward integration between spoken input and a text-based large language model (LLM). Wav2Prompt uses a simple training process with onl…
Parameter Efficient Finetuning for Speech Emotion Recognition and Domain Adaptation
Nineli Lashkarashvili, Wen Wu, Guangzhi Sun +1
Foundation models have shown superior performance for speech emotion recognition (SER). However, given the limited data in emotion corpora, finetuning all parameters of large pre-t…
Fine-grained Audio-Visual Joint Representations for Multimodal Large Language Models
Guangzhi Sun, Wenyi Yu, Changli Tang +6
Audio-visual large language models (LLM) have drawn significant attention, yet the fine-grained combination of both input streams is rather under-explored, which is challenging but…
Conditional Diffusion Model for Target Speaker Extraction
Theodor Nguyen, Guangzhi Sun, Xianrui Zheng +2
We propose DiffSpEx, a generative target speaker extraction method based on score-based generative modelling through stochastic differential equations. DiffSpEx deploys a continuou…
Connecting Speech Encoder and Large Language Model for ASR
Wenyi Yu, Changli Tang, Guangzhi Sun +6
The impressive capability and versatility of large language models (LLMs) have aroused increasing attention in automatic speech recognition (ASR), with several pioneering studies a…
Generating diverse and natural text-to-speech samples using a quantized fine-grained VAE and auto-regressive prosody prior
Guangzhi Sun, Yu Zhang, Ron J. Weiss +5
Recent neural text-to-speech (TTS) models with fine-grained latent features enable precise control of the prosody of synthesized speech. Such models typically incorporate a fine-gr…
Fully-hierarchical fine-grained prosody modeling for interpretable speech synthesis
Guangzhi Sun, Yu Zhang, Ron J. Weiss +3
This paper proposes a hierarchical, fine-grained and interpretable latent variable model for prosody based on the Tacotron 2 text-to-speech model. It achieves multi-resolution mode…