From the 1 of 13 linked papers with an AI index.
13 papers
Do LLMs Need Architectural Changes for Simultaneous Speech Translation? A Prefix-to-Prefix Data Driven Approach
Junkun Chen, Jian Xue, Ming Tang +4
The paper proposes a data‑driven prefix‑to‑prefix fine‑tuning method for simultaneous speech translation that works with decoder‑only LLMs without changing their architecture, usin…
Compress the Cache, Not the Speech Embedding: KV Compression for Efficient Speech LLMs
Ke-Han Lu, Keqi Deng, Ruchao Fan +2
Speech large language models (Speech LLMs) typically encode speech into sequences far longer than text, creating a major efficiency bottleneck during autoregressive decoding. A com…
Rethinking Speech-LLM Integration for ASR: Effective Joint Speech-Text Training by Interleaving
Ruchao Fan, Yiming Wang, Rui Zhao +10
Speech-LLM integration has shown promising results by leveraging extensive textual pretraining, yet its specific benefits for automatic speech recognition (ASR) remain unclear. We…
LLM can Read Spectrogram: Encoder-free Speech-Language Modeling
Ruchao Fan, Yiming Wang, Yuxuan Hu +6
Recent speech-aware large language models (Speech-LLMs) rely on pre-trained speech encoders to convert audio into semantic/acoustic rich representations consumable by LLM. In this…
Preserving Speech-to-Text LLM Capabilities in Speech-to-Speech Generation
Yuxuan Hu, Heng Lu, Ruchao Fan +8
Strong speech-to-text (S2T) LLMs already provide robust speech perception and text reasoning, but adding speech-to-speech (S2S) output is challenging: fine-tuning the backbone can…
UNIQUE: Universal Top-k Sparse Attention for Training-free Inference and Sparsity-aware Training
Keqi Deng, Shaoshi Ling, Ruchao Fan +1
Long-context inference in large language models (LLMs) is bottlenecked by the linear growth of the self-attention key-value (KV) cache. Top-k sparse attention alleviates this by lo…