20 citations · 100 across the 52 of their papers we have counts for
16 papers · 1 filter
OCR-Enhanced Multimodal ASR Can Read While Listening
Junli Chen, Changli Tang, Yixuan Li +2
Visual information, such as subtitles in a movie, often helps automatic speech recognition. In this paper, we propose Donut-Whisper, an audio-visual ASR model with dual encoder to…
Speech-Audio Compositional Attacks on Multimodal LLMs and Their Mitigation with SALMONN-Guard
Yudong Yang, Xuezhen Zhang, Zhifeng Han +6
Recent progress in LLMs has enabled understanding of audio signals, but has also exposed new safety risks arising from complex audio inputs that are inadequately handled by current…
Investigation of Zero-shot Text-to-Speech Models for Enhancing Short-Utterance Speaker Verification
Yiyang Zhao, Shuai Wang, Guangzhi Sun +4
Short-utterance speaker verification presents significant challenges due to the limited information in brief speech segments, which can undermine accuracy and reliability. Recently…
Low-Rank and Sparse Model Merging for Multi-Lingual Speech Recognition and Translation
Qiuming Zhao, Guangzhi Sun, Chao Zhang
Language diversity presents a significant challenge in speech-to-text (S2T) tasks, such as automatic speech recognition and translation. Traditional multi-lingual multi-task traini…
Whisper-PMFA: Partial Multi-Scale Feature Aggregation for Speaker Verification using Whisper Models
Yiyang Zhao, Shuai Wang, Guangzhi Sun +4
In this paper, Whisper, a large-scale pre-trained model for automatic speech recognition, is proposed to apply to speaker verification. A partial multi-scale feature aggregation (P…
Speaker Adaptation for Quantised End-to-End ASR Models
Qiuming Zhao, Guangzhi Sun, Chao Zhang +2
End-to-end models have shown superior performance for automatic speech recognition (ASR). However, such models are often very large in size and thus challenging to deploy on resour…
SAML: Speaker Adaptive Mixture of LoRA Experts for End-to-End ASR
Qiuming Zhao, Guangzhi Sun, Chao Zhang +2
Mixture-of-experts (MoE) models have achieved excellent results in many tasks. However, conventional MoE models are often very large, making them challenging to deploy on resource-…
Can Large Language Models Understand Spatial Audio?
Changli Tang, Wenyi Yu, Guangzhi Sun +8
This paper explores enabling large language models (LLMs) to understand spatial information from multichannel audio, a skill currently lacking in auditory LLMs. By leveraging LLMs'…
SALMONN: Towards Generic Hearing Abilities for Large Language Models
Changli Tang, Wenyi Yu, Guangzhi Sun +6
Hearing is arguably an essential ability of artificial intelligence (AI) agents in the physical world, which refers to the perception and understanding of general auditory informat…
Enhancing Quantised End-to-End ASR Models via Personalisation
Qiuming Zhao, Guangzhi Sun, Chao Zhang +2
Recent end-to-end automatic speech recognition (ASR) models have become increasingly larger, making them particularly challenging to be deployed on resource-constrained devices. Mo…
Cross-Utterance Conditioned VAE for Speech Generation
Yang Li, Cheng Yu, Guangzhi Sun +8
Speech synthesis systems powered by neural networks hold promise for multimedia production, but frequently face issues with producing expressive speech and seamless editing. In res…
Spectral Clustering-aware Learning of Embeddings for Speaker Diarisation
Evonne P. C. Lee, Guangzhi Sun, Chao Zhang +1
In speaker diarisation, speaker embedding extraction models often suffer from the mismatch between their training loss functions and the speaker clustering method. In this paper, w…
Tree-constrained Pointer Generator with Graph Neural Network Encodings for Contextual Speech Recognition
Guangzhi Sun, Chao Zhang, Philip C. Woodland
Incorporating biasing words obtained as contextual knowledge is critical for many automatic speech recognition (ASR) applications. This paper proposes the use of graph neural netwo…
Cross-Utterance Conditioned VAE for Non-Autoregressive Text-to-Speech
Yang Li, Cheng Yu, Guangzhi Sun +6
Modelling prosody variation is critical for synthesizing natural and expressive speech in end-to-end text-to-speech (TTS) systems. In this paper, a cross-utterance conditional VAE…
Content-Aware Speaker Embeddings for Speaker Diarisation
G. Sun, D. Liu, C. Zhang +1
Recent speaker diarisation systems often convert variable length speech segments into fixed-length vector representations for speaker clustering, which are known as speaker embeddi…
Combination of Deep Speaker Embeddings for Diarisation
Guangzhi Sun, Chao Zhang, Phil Woodland
Significant progress has recently been made in speaker diarisation after the introduction of d-vectors as speaker embeddings extracted from neural network (NN) speaker classifiers…