11 citations · 37 across the 26 of their papers we have counts for
30 papers
What Does LALM Post-training Learn? Rethinking Audio Understanding and Reasoning Capabilities
Hua-Hsuan Chen, Xuanjun Chen, Wei-Chung Lu +5
Post-training often improves the performance of large audio-language models on audio understanding and reasoning benchmarks. However, these improvements may reflect adaptation to s…
The NVVSpeech Challenge 2026: Overview and Lessons for Speech Understanding and Generation with Nonverbal Vocalizations
Liumeng Xue, Yuang Cao, Jingbin Hu +12
Nonverbal vocalizations (NVVs), such as laughter and sighs, convey meaning alongside words. NVV-aware speech systems must recognize and generate these events while preserving lexic…
EmoRES-TTS: Residual-Enhanced Vector Steering for Emotional Speech Generation
Kuan-Po Huang, Haohe Liu, Puyuan Peng +5
Emotion-conditioned text-to-speech (TTS) models may fail to express the requested emotion reliably, and improving controllability by additional training is costly in both computati…
Mitigating Proxy-to-Wild Domain Gap in Deepfake Speech
Xuanjun Chen, Yun-Shing Wu, Wei-Chung Lu +4
Recent neural audio codec-based speech generation (CodecFake) produces highly realistic audio, posing a challenge to existing deepfake countermeasure models. While using codec resy…
Joint Fullband-Subband Modeling for High-Resolution SingFake Detection
Xuanjun Chen, Chia-Yu Hu, Sung-Feng Huang +3
Rapid advances in singing voice synthesis have increased unauthorized imitation risks, creating an urgent need for better Singing Voice Deepfake (SingFake) Detection, also known as…
How Does Instrumental Music Help SingFake Detection?
Xuanjun Chen, Chia-Yu Hu, I-Ming Lin +8
Although many models exist to detect singing voice deepfakes (SingFake), how these models operate, particularly with instrumental accompaniment, is unclear. We investigate how inst…
Localizing Audio-Visual Deepfakes via Hierarchical Boundary Modeling
Xuanjun Chen, Shih-Peng Cheng, Jiawei Du +6
Audio-visual temporal deepfake localization under the content-driven partial manipulation remains a highly challenging task. In this scenario, the deepfake regions are usually only…
HighRateMOS: Sampling-Rate Aware Modeling for Speech Quality Assessment
Wenze Ren, Yi-Cheng Lin, Wen-Chin Huang +9
Modern speech quality prediction models are trained on audio data resampled to a specific sampling rate. When faced with higher-rate audio at test time, these models can produce bi…
Towards Generalized Source Tracing for Codec-Based Deepfake Speech
Xuanjun Chen, I-Ming Lin, Lin Zhang +3
Recent attempts at source tracing for codec-based deepfake speech (CodecFake), generated by neural audio codec-based speech generation (CoSG) models, have exhibited suboptimal perf…
Codec-Based Deepfake Source Tracing via Neural Audio Codec Taxonomy
Xuanjun Chen, I-Ming Lin, Lin Zhang +4
Recent advances in neural audio codec-based speech generation (CoSG) models have produced remarkably realistic audio deepfakes. We refer to deepfake speech generated by CoSG system…
CodecFake+: Codec-Based Resynthesized Data as a Proxy for Detecting CodecFake Speech
Xuanjun Chen, Jiawei Du, Haibin Wu +8
With the rapid advancement of neural audio codecs, codec-based speech generation (CoSG) systems have become highly powerful. Unfortunately, CoSG also enables the creation of highly…
Dynamic-SUPERB Phase-2: A Collaboratively Expanding Benchmark for Measuring the Capabilities of Spoken Language Models with 180 Tasks
Chien-yu Huang, Wei-Chih Chen, Shu-wen Yang +77
Multimodal foundation models, such as Gemini and ChatGPT, have revolutionized human-machine interactions by seamlessly integrating various forms of data. Developing a universal spo…
Codec-SUPERB @ SLT 2024: A lightweight benchmark for neural audio codec models
Haibin Wu, Xuanjun Chen, Yi-Cheng Lin +13
Neural audio codec models are becoming increasingly important as they serve as tokenizers for audio, enabling efficient transmission or facilitating speech language modeling. The i…
DFADD: The Diffusion and Flow-Matching Based Audio Deepfake Dataset
Jiawei Du, I-Ming Lin, I-Hsiang Chiu +6
Mainstream zero-shot TTS production systems like Voicebox and Seed-TTS achieve human parity speech by leveraging Flow-matching and Diffusion models, respectively. Unfortunately, hu…
Efficient Training of Self-Supervised Speech Foundation Models on a Compute Budget
Andy T. Liu, Yi-Cheng Lin, Haibin Wu +2
Despite their impressive success, training foundation models remains computationally costly. This paper investigates how to efficiently train speech foundation models with self-sup…
ESPnet-Codec: Comprehensive Training and Evaluation of Neural Codecs for Audio, Music, and Speech
Jiatong Shi, Jinchuan Tian, Yihan Wu +17
Neural codecs have become crucial to recent speech and audio generation research. In addition to signal compression capabilities, discrete codecs have also been found to enhance do…
Stimulus Modality Matters: Impact of Perceptual Evaluations from Different Modalities on Speech Emotion Recognition System Performance
Huang-Cheng Chou, Haibin Wu, Hung-yi Lee +1
Speech Emotion Recognition (SER) systems rely on speech input and emotional labels annotated by humans. However, various emotion databases collect perceptional evaluations in diffe…
Leveraging Joint Spectral and Spatial Learning with MAMBA for Multichannel Speech Enhancement
Wenze Ren, Haibin Wu, Yi-Cheng Lin +7
In multichannel speech enhancement, effectively capturing spatial and spectral information across different microphones is crucial for noise reduction. Traditional methods, such as…
SpeechPrompt: Prompting Speech Language Models for Speech Processing Tasks
Kai-Wei Chang, Haibin Wu, Yu-Kai Wang +6
Prompting has become a practical method for utilizing pre-trained language models (LMs). This approach offers several advantages. It allows an LM to adapt to new tasks with minimal…
EMO-Codec: An In-Depth Look at Emotion Preservation capacity of Legacy and Neural Codec Models With Subjective and Objective Evaluations
Wenze Ren, Yi-Cheng Lin, Huang-Cheng Chou +5
The neural codec model reduces speech data transmission delay and serves as the foundational tokenizer for speech language models (speech LMs). Preserving emotional information in…
CodecFake: Enhancing Anti-Spoofing Models Against Deepfake Audios from Codec-Based Speech Synthesis Systems
Haibin Wu, Yuan Tseng, Hung-yi Lee
Current state-of-the-art (SOTA) codec-based audio synthesis systems can mimic anyone's voice with just a 3-second sample from that specific unseen speaker. Unfortunately, malicious…
Singing Voice Graph Modeling for SingFake Detection
Xuanjun Chen, Haibin Wu, Jyh-Shing Roger Jang +1
Detecting singing voice deepfakes, or SingFake, involves determining the authenticity and copyright of a singing voice. Existing models for speech deepfake detection have struggled…
Neural Codec-based Adversarial Sample Detection for Speaker Verification
Xuanjun Chen, Jiawei Du, Haibin Wu +2
Automatic Speaker Verification (ASV), increasingly used in security-critical applications, faces vulnerabilities from rising adversarial attacks, with few effective defenses availa…
Emo-bias: A Large Scale Evaluation of Social Bias on Speech Emotion Recognition
Yi-Cheng Lin, Haibin Wu, Huang-Cheng Chou +2
The rapid growth of Speech Emotion Recognition (SER) has diverse global applications, from improving human-computer interactions to aiding mental health diagnostics. However, SER m…