8 citations · 20 across the 23 of their papers we have counts for
26 papers
D2: High-Capability Full-Duplex Spoken Language Models from Half-Duplex Backbones
Cheng-Kuang Chang, Alexander H. Liu, Brian Kingsbury +5
Full-duplex spoken language models (FD-SLMs) support simultaneous listening and speaking, but often lag behind half-duplex spoken language models (HD-SLMs) in general capabilities.…
MP-Bench: Evaluating Voice Agents as a Multiparty Conversation Participant
Yi-Jen Shih, Shih-Yun Shan Kuan, Guan-Ting Lin +7
Conversational voice agents have advanced significantly, offering increasingly natural human-machine interactions through both cascaded and end-to-end architectures. However, while…
AudioICL-Bench: A Benchmark for Large Audio Language Model In-Context Learning
Jia-Hung Chen, Yi-Cheng Lin, Kai-Wei Chang +2
In-context learning (ICL) promises training-free adaptation for audio, where labeling every new condition is costly. Yet existing audio ICL studies largely measure Task Recognition…
Unsupervised Speech Recognition at the Syllable Level
Liming Wang, Kai-Wei Chang, Kunio Kashino +3
Training speech recognizers with unpaired speech and text -- known as unsupervised speech recognition (UASR) -- is a crucial step toward extending ASR to low-resource languages in…
Multimodal Unlearning Across Vision, Language, Video, and Audio: Survey of Methods, Datasets, and Benchmarks
Nobin Sarwar, Shubhashis Roy Dipta, Zheyuan Liu +1
With the growing adoption of VLMs, DMs, LLMs, and AFMs, these multimodal foundation models can inadvertently encode sensitive, copyrighted, biased, or unsafe cross-modal associatio…
Overcoming State Inertia in Full-Duplex Spoken Language Models via Activation Steering
Cheng-Kuang Chang, Kai-Wei Chang, Alexander H. Liu +1
Full-duplex spoken language models (FD-SLMs) enable seamless speech interaction by allowing models to listen and speak simultaneously, yet the internal mechanism by which they coor…
RRP-Voice: A Longitudinal Dataset and Benchmark for Recurrent Respiratory Papillomatosis Detection
Wenze Ren, Ke-Han Lu, Kai-Wei Chang +8
Deep learning has advanced pathological voice detection rapidly, yet rare laryngeal diseases remain underexplored due to data scarcity. Recurrent Respiratory Papillomatosis (RRP) e…
TiCo: Time-Controllable Spoken Dialogue Model
Kai-Wei Chang, Wei-Chih Chen, En-Pei Hu +2
We introduce TiCo, a time-controllable spoken dialogue model (SDM) that follows time-constrained instructions (e.g., "Please generate a response lasting about 15 seconds") and gene…
TaigiSpeech: A Low-Resource Real-World Speech Intent Dataset and Preliminary Results with Scalable Data Mining In-the-Wild
Kai-Wei Chang, Yi-Cheng Lin, Huang-Cheng Chou +9
Speech technologies have advanced rapidly and serve diverse populations worldwide. However, many languages remain underrepresented due to limited resources. In this paper, we intro…
AQAScore: Evaluating Semantic Alignment in Text-to-Audio Generation via Audio Question Answering
Chun-Yi Kuan, Kai-Wei Chang, Hung-yi Lee
Although text-to-audio generation has made remarkable progress in realism and diversity, the development of evaluation metrics has not kept pace. Widely-adopted approaches, typical…
On the Fallacy of Global Token Perplexity in Spoken Language Model Evaluation
Chan-Jan Hsu, Liang-Hsuan Tseng, Yi-Cheng Lin +5
Generative spoken language models pretrained on large-scale raw audio can continue a speech prompt with appropriate content while preserving attributes like speaker and emotion, se…
BILLY: Steering Large Language Models via Merging Persona Vectors for Creative Generation
Tsung-Min Pai, Jui-I Wang, Li-Chun Lu +3
Multi-LLM systems enhance the creativity of large language models by simulating human collective intelligence but suffer from significant drawbacks, such as high computational cost…
Full-Duplex-Bench-v2: A Multi-Turn Evaluation Framework for Duplex Dialogue Systems with an Automated Examiner
Guan-Ting Lin, Shih-Yun Shan Kuan, Jiatong Shi +4
While full-duplex speech agents enable natural, low-latency interaction by speaking and listening simultaneously, their consistency and task performance in multi-turn settings rema…
Game-Time: Evaluating Temporal Dynamics in Spoken Language Models
Kai-Wei Chang, En-Pei Hu, Chun-Yi Kuan +7
Conversational Spoken Language Models (SLMs) are emerging as a promising paradigm for real-time speech interaction. However, their capacity of temporal dynamics, including the abil…
DeSTA2.5-Audio: Toward General-Purpose Large Audio Language Model with Self-Generated Cross-Modal Alignment
Ke-Han Lu, Zhehuai Chen, Szu-Wei Fu +25
We introduce DeSTA2.5-Audio, a general-purpose Large Audio Language Model (LALM) designed for robust auditory perception and instruction-following. Recent LALMs augment Large Langu…
Analyzing Mitigation Strategies for Catastrophic Forgetting in End-to-End Training of Spoken Language Models
Chi-Yuan Hsiao, Ke-Han Lu, Kai-Wei Chang +3
End-to-end training of Spoken Language Models (SLMs) commonly involves adapting pre-trained text-based Large Language Models (LLMs) to the speech modality through multi-stage train…
SIFT-50M: A Large-Scale Multilingual Dataset for Speech Instruction Fine-Tuning
Prabhat Pandey, Rupak Vignesh Swaminathan, K V Vijay Girish +4
We introduce SIFT (Speech Instruction Fine-Tuning), a 50M-example dataset designed for instruction fine-tuning and pre-training of speech-text large language models (LLMs). SIFT-50…
On The Landscape of Spoken Language Models: A Comprehensive Survey
Siddhant Arora, Kai-Wei Chang, Chung-Ming Chien +7
The field of spoken language processing is undergoing a shift from training custom-built, task-specific models toward using and optimizing spoken language models (SLMs) which act a…
Dynamic-SUPERB Phase-2: A Collaboratively Expanding Benchmark for Measuring the Capabilities of Spoken Language Models with 180 Tasks
Chien-yu Huang, Wei-Chih Chen, Shu-wen Yang +77
Multimodal foundation models, such as Gemini and ChatGPT, have revolutionized human-machine interactions by seamlessly integrating various forms of data. Developing a universal spo…
Codec-SUPERB @ SLT 2024: A lightweight benchmark for neural audio codec models
Haibin Wu, Xuanjun Chen, Yi-Cheng Lin +13
Neural audio codec models are becoming increasingly important as they serve as tokenizers for audio, enabling efficient transmission or facilitating speech language modeling. The i…
SpeechPrompt: Prompting Speech Language Models for Speech Processing Tasks
Kai-Wei Chang, Haibin Wu, Yu-Kai Wang +6
Prompting has become a practical method for utilizing pre-trained language models (LMs). This approach offers several advantages. It allows an LM to adapt to new tasks with minimal…
EMO-SUPERB: An In-depth Look at Speech Emotion Recognition
Haibin Wu, Huang-Cheng Chou, Kai-Wei Chang +5
Speech emotion recognition (SER) is a pivotal technology for human-computer interaction systems. However, 80.77% of SER papers yield results that cannot be reproduced. We develop E…
Towards audio language modeling -- an overview
Haibin Wu, Xuanjun Chen, Yi-Cheng Lin +4
Neural audio codecs are initially introduced to compress audio data into compact codes to reduce transmission latency. Researchers recently discovered the potential of codecs as su…
Codec-SUPERB: An In-Depth Analysis of Sound Codec Models
Haibin Wu, Ho-Lam Chung, Yi-Cheng Lin +7
The sound codec's dual roles in minimizing data transmission latency and serving as tokenizers underscore its critical importance. Recent years have witnessed significant developme…