10 papers · 1 filter
Continual Speech Learning with Fused Speech Features
Guitao Wang, Jinming Zhao, Hao Yang +3
Rapid growth in speech data demands adaptive models, as traditional static methods fail to keep pace with dynamic and diverse speech information. We introduce continuous speech lea…
Reshaping Representation Space to Balance the Safety and Over-rejection in Large Audio Language Models
Hao Yang, Lizhen Qu, Ehsan Shareghi +1
Large Audio Language Models (LALMs) have extended the capabilities of Large Language Models (LLMs) by enabling audio-based human interactions. However, recent research has revealed…
Audio Is the Achilles' Heel: Red Teaming Audio Large Multimodal Models
Hao Yang, Lizhen Qu, Ehsan Shareghi +1
Large Multimodal Models (LMMs) have demonstrated the ability to interact with humans under real-world conditions by combining Large Language Models (LLMs) and modality encoders to…
Jigsaw Puzzles: Splitting Harmful Questions to Jailbreak Large Language Models
Hao Yang, Lizhen Qu, Ehsan Shareghi +1
Large language models (LLMs) have exhibited outstanding performance in engaging with humans and addressing complex questions by leveraging their vast implicit knowledge and robust…
Towards Probing Speech-Specific Risks in Large Multimodal Models: A Taxonomy, Benchmark, and Insights
Hao Yang, Lizhen Qu, Ehsan Shareghi +1
Large Multimodal Models (LMMs) have achieved great success recently, demonstrating a strong capability to understand multimodal information and to interact with human users. Despit…
Double Mixture: Towards Continual Event Detection from Speech
Jingqi Kang, Tongtong Wu, Jinming Zhao +6
Speech event detection is crucial for multimedia retrieval, involving the tagging of both semantic and acoustic events. Traditional ASR systems often overlook the interplay between…
Investigating Pre-trained Audio Encoders in the Low-Resource Condition
Hao Yang, Jinming Zhao, Gholamreza Haffari +1
Pre-trained speech encoders have been central to pushing state-of-the-art results across various speech understanding and generation tasks. Nonetheless, the capabilities of these e…
Self-supervised Rewiring of Pre-trained Speech Encoders: Towards Faster Fine-tuning with Less Labels in Speech Processing
Hao Yang, Jinming Zhao, Gholamreza Haffari +1
Pre-trained speech Transformers have facilitated great success across various speech processing tasks. However, fine-tuning these encoders for downstream tasks require sufficiently…
RedApt: An Adaptor for wav2vec 2 Encoding \\ Faster and Smaller Speech Translation without Quality Compromise
Jinming Zhao, Hao Yang, Gholamreza Haffari +1
Pre-trained speech Transformers in speech translation (ST) have facilitated state-of-the-art (SotA) results; yet, using such encoders is computationally expensive. To improve this,…
M-Adapter: Modality Adaptation for End-to-End Speech-to-Text Translation
Jinming Zhao, Hao Yang, Ehsan Shareghi +1
End-to-end speech-to-text translation models are often initialized with pre-trained speech encoder and pre-trained text decoder. This leads to a significant training gap between pr…