5 papers
Text-Prompted CLAP: Learning Query-Conditioned Audio Representations via Contrastive Learning
Mohan Li, Rama Doddipatla, Philip C. Woodland
Contrastive Language-Audio Pretraining (CLAP) learns aligned text and audio representations in a shared embedding space. However, independent encoding of each modality limits its a…
SALMONN-2: Advancing General-Purpose Hearing Abilities with Self-Supervised Representations
Xiaoyu Yang, Xuenan Xu, Wenyi Yu +10
Recent audio large language models (ALLMs) are typically built upon audio encoders trained with large amounts of supervised data. Since self-supervised learning (SSL) audio encoder…
Multi-head Temporal Latent Attention
Keqi Deng, Philip C. Woodland
While Transformer self-attention offers strong parallelism, the Key-Value (KV) cache grows linearly with sequence length and becomes a bottleneck for inference efficiency. Multi-he…
SimulS2S-LLM: Unlocking Simultaneous Inference of Speech LLMs for Speech-to-Speech Translation
Keqi Deng, Wenxi Chen, Xie Chen +1
Simultaneous speech translation (SST) outputs translations in parallel with streaming speech input, balancing translation quality and latency. While large language models (LLMs) ha…
Transducer-Llama: Integrating LLMs into Streamable Transducer-based Speech Recognition
Keqi Deng, Jinxi Guo, Yingyi Ma +4
While large language models (LLMs) have been applied to automatic speech recognition (ASR), the task of making the model streamable remains a challenge. This paper proposes a novel…