13 papers
Emo-LiPO: Listwise Preference Optimization for Fine-Grained Emotion Intensity Control in LLM-based Text-to-Speech
Yihang Lin, Li Zhou, Congwei Cao +4
Large language model (LLM)-based text-to-speech (TTS) systems enable prompt-conditioned emotional control but struggle with fine-grained emotion intensity due to the semantic -- ac…
PAL: Prompting Analytic Learning with Missing Modality for Multi-Modal Class-Incremental Learning
Xianghu Yue, Yiming Chen, Xueyi Zhang +5
Multi-modal class-incremental learning (MMCIL) seeks to leverage multi-modal data, such as audio-visual and image-text pairs, thereby enabling models to learn continuously across a…
Transferable Adversarial Attacks against ASR
Xiaoxue Gao, Zexin Li, Yiming Chen +2
Given the extensive research and real-world applications of automatic speech recognition (ASR), ensuring the robustness of ASR models against minor input perturbations becomes a cr…
VoiceBench: Benchmarking LLM-Based Voice Assistants
Yiming Chen, Xianghu Yue, Chen Zhang +3
Building on the success of large language models (LLMs), recent advancements such as GPT-4o have enabled real-time speech interactions through LLM-based voice assistants, offering…
Beyond Single-Audio: Advancing Multi-Audio Processing in Audio Large Language Models
Yiming Chen, Xianghu Yue, Xiaoxue Gao +4
Various audio-LLMs (ALLMs) have been explored recently for tackling different audio tasks simultaneously using a single, unified model. While existing evaluations of ALLMs primaril…
Enhancing Real-World Active Speaker Detection with Multi-Modal Extraction Pre-Training
Ruijie Tao, Xinyuan Qian, Rohan Kumar Das +3
Audio-visual active speaker detection (AV-ASD) aims to identify which visible face is speaking in a scene with one or more persons. Most existing AV-ASD methods prioritize capturin…
Text-guided HuBERT: Self-Supervised Speech Pre-training via Generative Adversarial Networks
Duo Ma, Xianghu Yue, Junyi Ao +2
Human language can be expressed in either written or spoken form, i.e. text or speech. Humans can acquire knowledge from text to improve speaking and listening. However, the quest…
Self-Transriber: Few-shot Lyrics Transcription with Self-training
Xiaoxue Gao, Xianghu Yue, Haizhou Li
The current lyrics transcription approaches heavily rely on supervised learning with labeled data, but such data are scarce and manual labeling of singing is expensive. How to bene…
token2vec: A Joint Self-Supervised Pre-training Framework Using Unpaired Speech and Text
Xianghu Yue, Junyi Ao, Xiaoxue Gao +1
Self-supervised pre-training has been successful in both text and speech processing. Speech and text offer different but complementary information. The question is whether we are a…
PoLyScriber: Integrated Fine-tuning of Extractor and Lyrics Transcriber for Polyphonic Music
Xiaoxue Gao, Chitralekha Gupta, Haizhou Li
Lyrics transcription of polyphonic music is challenging as the background music affects lyrics intelligibility. Typically, lyrics transcription can be performed by a two-step pipel…
Music-robust Automatic Lyrics Transcription of Polyphonic Music
Xiaoxue Gao, Chitralekha Gupta, Haizhou Li
Lyrics transcription of polyphonic music is challenging because singing vocals are corrupted by the background music. To improve the robustness of lyrics transcription to the backg…
Genre-conditioned Acoustic Models for Automatic Lyrics Transcription of Polyphonic Music
Xiaoxue Gao, Chitralekha Gupta, Haizhou Li
Lyrics transcription of polyphonic music is challenging not only because the singing vocals are corrupted by the background music, but also because the background music and the sin…
NHSS: A Speech and Singing Parallel Database
Bidisha Sharma, Xiaoxue Gao, Karthika Vijayan +2
We present a database of parallel recordings of speech and singing, collected and released by the Human Language Technology (HLT) laboratory at the National University of Singapore…