216 citations · 556 across the 81 of their papers we have counts for
13 papers · 1 filter
Encoder-Free Multimodal LLMs Learn Implicit Encoders
David Harwath, Desh Raj, Xavier Velez +5
We empirically analyze two recently proposed encoder-free multimodal LLMs, Gemma-4 12B and Inkling-Small. Unlike previous multimodal LLMs, these models do not use modality-specific…
ASSET: Adaptive Speech Segmentation via Reinforcement Learning
Cihan Xiao, Yuan Tseng, Matthew Wiesner +4
Speech large language models (LLMs) are often built using pretrained acoustic encoders as an audio interface to an LLM decoder. However, LLMs scale poorly with input sequence lengt…
All In Good Time: Causality-Aware Framework for LLM-Based Simultaneous Speech-to-Speech Translation
Amir Hussein, Enas Albasiri, Travis M. Bartley +7
Large Language Models (LLMs) have shown strong performance in low-resource offline translation; however, extending them to simultaneous speech-to-speech translation (Simul-S2ST) re…
LAYERSCOPE: A Layerwise Characterization of Video and Multimodal Learned Representations
Sandra Arcos-Holzinger, Debashish Chakraborty, Rohita Mocharla +7
We propose LAYERSCOPE, a label-free, layerwise framework that aims to characterize a model's learned representations in video and multimodal settings. Evaluating downstream perform…
Clustering Unsupervised Representations as Defense against Poisoning Attacks on Speech Commands Classification System
Thomas Thebaud, Sonal Joshi, Henry Li +4
Poisoning attacks entail attackers intentionally tampering with training data. In this paper, we consider a dirty-label poisoning attack scenario on a speech commands classificatio…
Escape the Language Prior: Mitigating Late-Stage Modality Collapse in Audio Reasoning via Modality-Aware Policy Optimization
Cihan Xiao, Yiwen Shao, Chenxing Li +5
Audio and omni-modal large language models exhibit impressive cross-modal reasoning capabilities. However, standard reinforcement learning methods such as GRPO do not explicitly we…
Dimensionality-Aware Anomaly Detection in Learned Representations of Self-Supervised Speech Models
Sandra Arcos-Holzinger, Sarah M. Erfani, James Bailey +1
Self-supervised speech models (S3Ms) achieve strong downstream performance, yet their learned representations remain poorly understood under natural and adversarial perturbations.…
Modeling Overlapped Speech with Shuffles
Matthew Wiesner, Samuele Cornell, Alexander Polok +3
We propose to model parallel streams of data, such as overlapped speech, using shuffles. Specifically, this paper shows how the shuffle product and partial order finite-state autom…
Integrated Spoofing-Robust Automatic Speaker Verification via a Three-Class Formulation and LLR
Kai Tan, Lin Zhang, Ruiteng Zhang +6
Spoofing-robust automatic speaker verification (SASV) aims to integrate automatic speaker verification (ASV) and countermeasure (CM). A popular solution is fusion of independent AS…
Can LLMs Help Localize Fake Words in Partially Fake Speech?
Lin Zhang, Thomas Thebaud, Zexin Cai +5
Large language models (LLMs), trained on large-scale text, have recently attracted significant attention for their strong performance across many tasks. Motivated by this, we inves…
Universal Speech Content Factorization
Henry Li Xinyuan, Zexin Cai, Lin Zhang +5
We propose Universal Speech Content Factorization (USCF), a simple and invertible linear method for extracting a low-rank speech representation in which speaker timbre is suppresse…
SE-DiCoW: Self-Enrolled Diarization-Conditioned Whisper
Alexander Polok, Dominik Klement, Samuele Cornell +4
Speaker-attributed automatic speech recognition (ASR) in multi-speaker environments remains a major challenge. While some approaches achieve strong performance when fine-tuned on s…
SpatialEmb: Extract and Encode Spatial Information for 1-Stage Multi-channel Multi-speaker ASR on Arbitrary Microphone Arrays
Yiwen Shao, Yong Xu, Sanjeev Khudanpur +1
Spatial information is a critical clue for multi-channel multi-speaker target speech recognition. Most state-of-the-art multi-channel Automatic Speech Recognition (ASR) systems ext…