6 citations · 16 across the 19 of their papers we have counts for
21 papers
Teaching LLMs to Hear Who Spoke What: Metadata-Supervised Pretraining for Encoder-Free Speech-LLMs
Mohan Shi, Ruchao Fan, Sunit Sivasankaran +2
Encoder-based speech large language models (Speech-LLMs) commonly employ pretrained speech encoders that prioritize linguistic content but may discard fine-grained acoustic cues es…
Compress the Cache, Not the Speech Embedding: KV Compression for Efficient Speech LLMs
Ke-Han Lu, Keqi Deng, Ruchao Fan +2
Speech large language models (Speech LLMs) typically encode speech into sequences far longer than text, creating a major efficiency bottleneck during autoregressive decoding. A com…
Rethinking Speech-LLM Integration for ASR: Effective Joint Speech-Text Training by Interleaving
Ruchao Fan, Yiming Wang, Rui Zhao +10
Speech-LLM integration has shown promising results by leveraging extensive textual pretraining, yet its specific benefits for automatic speech recognition (ASR) remain unclear. We…
OpenSTBench: Beyond Semantic Evaluation for Speech Translation
Yanjie An, Yuxiang Zhao, Yichi Zhang +5
Speech translation systems increasingly span speech-to-text translation (S2TT), speech-to-speech translation (S2ST), offline translation, and streaming generation, producing output…
UNIQUE: Universal Top-k Sparse Attention for Training-free Inference and Sparsity-aware Training
Keqi Deng, Shaoshi Ling, Ruchao Fan +1
Long-context inference in large language models (LLMs) is bottlenecked by the linear growth of the self-attention key-value (KV) cache. Top-k sparse attention alleviates this by lo…
Speech LLMs are Contextual Reasoning Transcribers
Keqi Deng, Ruchao Fan, Bo Ren +2
Despite extensions to speech inputs, effectively leveraging the rich knowledge and contextual understanding of large language models (LLMs) in automatic speech recognition (ASR) re…
Multi-head Temporal Latent Attention
Keqi Deng, Philip C. Woodland
While Transformer self-attention offers strong parallelism, the Key-Value (KV) cache grows linearly with sequence length and becomes a bottleneck for inference efficiency. Multi-he…
Transducer-Llama: Integrating LLMs into Streamable Transducer-based Speech Recognition
Keqi Deng, Jinxi Guo, Yingyi Ma +4
While large language models (LLMs) have been applied to automatic speech recognition (ASR), the task of making the model streamable remains a challenge. This paper proposes a novel…
Label-Synchronous Neural Transducer for E2E Simultaneous Speech Translation
Keqi Deng, Philip C. Woodland
While the neural transducer is popular for online speech recognition, simultaneous speech translation (SST) requires both streaming and re-ordering capabilities. This paper present…
Wav2Prompt: End-to-End Speech Prompt Generation and Tuning For LLM in Zero and Few-shot Learning
Keqi Deng, Guangzhi Sun, Philip C. Woodland
Wav2Prompt is proposed which allows straightforward integration between spoken input and a text-based large language model (LLM). Wav2Prompt uses a simple training process with onl…
FastInject: Injecting Unpaired Text Data into CTC-based ASR training
Keqi Deng, Philip C. Woodland
Recently, connectionist temporal classification (CTC)-based end-to-end (E2E) automatic speech recognition (ASR) models have achieved impressive results, especially with the develop…
Label-Synchronous Neural Transducer for Adaptable Online E2E Speech Recognition
Keqi Deng, Philip C. Woodland
Although end-to-end (E2E) automatic speech recognition (ASR) has shown state-of-the-art recognition accuracy, it tends to be implicitly biased towards the training data distributio…
Decoupled Structure for Improved Adaptability of End-to-End Models
Keqi Deng, Philip C. Woodland
Although end-to-end (E2E) trainable automatic speech recognition (ASR) has shown great success by jointly learning acoustic and linguistic information, it still suffers from the ef…
Label-Synchronous Neural Transducer for End-to-End ASR
Keqi Deng, Philip C. Woodland
Neural transducers provide a natural way of streaming ASR. However, they augment output sequences with blank tokens which leads to challenges for domain adaptation using text data.…
Adaptable End-to-End ASR Models using Replaceable Internal LMs and Residual Softmax
Keqi Deng, Philip C. Woodland
End-to-end (E2E) automatic speech recognition (ASR) implicitly learns the token sequence distribution of paired audio-transcript training data. However, it still suffers from domai…
Improving Streaming End-to-End ASR on Transformer-based Causal Models with Encoder States Revision Strategies
Zehan Li, Haoran Miao, Keqi Deng +4
There is often a trade-off between performance and latency in streaming automatic speech recognition (ASR). Traditional methods such as look-ahead and chunk-based methods, usually…
Blockwise Streaming Transformer for Spoken Language Understanding and Simultaneous Speech Translation
Keqi Deng, Shinji Watanabe, Jiatong Shi +1
Although Transformers have gained success in several speech processing tasks like spoken language understanding (SLU) and speech translation (ST), achieving online processing while…
Improving CTC-based speech recognition via knowledge transferring from pre-trained language models
Keqi Deng, Songjun Cao, Yike Zhang +4
Recently, end-to-end automatic speech recognition models based on connectionist temporal classification (CTC) have achieved impressive results, especially when fine-tuned from wav2…
Improving non-autoregressive end-to-end speech recognition with pre-trained acoustic and language models
Keqi Deng, Zehui Yang, Shinji Watanabe +3
While Transformers have achieved promising results in end-to-end (E2E) automatic speech recognition (ASR), their autoregressive (AR) structure becomes a bottleneck for speeding up…
Improving Hybrid CTC/Attention End-to-end Speech Recognition with Pretrained Acoustic and Language Model
Keqi Deng, Songjun Cao, Yike Zhang +1
Recently, self-supervised pretraining has achieved impressive results in end-to-end (E2E) automatic speech recognition (ASR). However, the dominant sequence-to-sequence (S2S) E2E m…
Improving Accent Identification and Accented Speech Recognition Under a Framework of Self-supervised Learning
Keqi Deng, Songjun Cao, Long Ma
Recently, self-supervised pre-training has gained success in automatic speech recognition (ASR). However, considering the difference between speech accents in real scenarios, how t…