12 papers
Escaping the Procrustean Bed: Groupwise Orthogonal Connectors for Audio-Language Models
Ho-Lam Chung, Ke-Han Lu, Yi-Cheng Lin +3
Audio-language models compress a speech encoder's output through a Querying Transformer (Q-Former) connector before feeding it to a large language model. We identify two failures i…
Context-Aware ASR for Mandarin Technical Lectures
Ho-Lam Chung, Yiming Chen, Hung-yi Lee
Technical lectures mix Mandarin speech with English technical terms. These terms carry the core meaning of the lecture, yet they occupy few characters. Character error rate (CER) t…
Listen, Think, Transcribe: Continuous Latent Test-Time Scaling for ASR
Ho Lam Chung, Yiming Chen, Dau-Cheng Lyu +2
End-to-end ASR models transcribe in a single pass, leaving no room for the decoder to revisit hard inputs. We propose LatentASR, a parameter-efficient method that adds continuous l…
HalluAudio: A Comprehensive Benchmark for Hallucination Detection in Large Audio-Language Models
Feiyu Zhao, Yiming Chen, Wenhuan Lu +3
Large Audio-Language Models (LALMs) have recently achieved strong performance across various audio-centric tasks. However, hallucination, where models generate responses that are s…
LLM-Codec: Neural Audio Codec Meets Language Model Objectives
Ho-Lam Chung, Yiming Chen, Hung-yi Lee
Neural audio codecs are widely used as tokenizers for spoken language models, but they are optimized for waveform reconstruction rather than autoregressive prediction. This mismatc…
MORE: Multi-Objective Adversarial Attacks on Speech Recognition
Xiaoxue Gao, Zexin Li, Yiming Chen +1
The emergence of large-scale automatic speech recognition (ASR) models such as Whisper has greatly expanded their adoption across diverse real-world applications. Ensuring robustne…
PAL: Prompting Analytic Learning with Missing Modality for Multi-Modal Class-Incremental Learning
Xianghu Yue, Yiming Chen, Xueyi Zhang +5
Multi-modal class-incremental learning (MMCIL) seeks to leverage multi-modal data, such as audio-visual and image-text pairs, thereby enabling models to learn continuously across a…
Transferable Adversarial Attacks against ASR
Xiaoxue Gao, Zexin Li, Yiming Chen +2
Given the extensive research and real-world applications of automatic speech recognition (ASR), ensuring the robustness of ASR models against minor input perturbations becomes a cr…
VoiceBench: Benchmarking LLM-Based Voice Assistants
Yiming Chen, Xianghu Yue, Chen Zhang +3
Building on the success of large language models (LLMs), recent advancements such as GPT-4o have enabled real-time speech interactions through LLM-based voice assistants, offering…
Emo-DPO: Controllable Emotional Speech Synthesis through Direct Preference Optimization
Xiaoxue Gao, Chen Zhang, Yiming Chen +2
Current emotional text-to-speech (TTS) models predominantly conduct supervised training to learn the conversion from text and desired emotion to its emotional speech, focusing on a…
Beyond Single-Audio: Advancing Multi-Audio Processing in Audio Large Language Models
Yiming Chen, Xianghu Yue, Xiaoxue Gao +4
Various audio-LLMs (ALLMs) have been explored recently for tackling different audio tasks simultaneously using a single, unified model. While existing evaluations of ALLMs primaril…
TTSlow: Slow Down Text-to-Speech with Efficiency Robustness Evaluations
Xiaoxue Gao, Yiming Chen, Xianghu Yue +2
Text-to-speech (TTS) has been extensively studied for generating high-quality speech with textual inputs, playing a crucial role in various real-time applications. For real-world d…