6 papers
Double-P: Hierarchical Top-P Sparse Attention for Long-Context LLMs
Wentao Ni, Kangqi Zhang, Zhongming Yu +7
As long-context inference becomes central to large language models (LLMs), attention over growing key-value caches emerges as a dominant decoding bottleneck, motivating sparse atte…
Fast Forward: Accelerating LLM Prefill with Predictive FFN Sparsity
Aayush Gautam, Mukul Gagrani, Junyoung Park +3
The prefill stage of large language model (LLM) inference is a key computational bottleneck for long-context workloads. At short-to-moderate context lengths (1K--16K tokens), Feed-…
VOCABTRIM: Vocabulary Pruning for Efficient Speculative Decoding in LLMs
Raghavv Goel, Sudhanshu Agrawal, Mukul Gagrani +9
In this paper, we introduce a simple training-free technique to improve the performance of drafter-based speculative decoding (SpD) methods that incorporates language modeling head…
KeyDiff: Key Similarity-Based KV Cache Eviction for Long-Context LLM Inference in Resource-Constrained Environments
Junyoung Park, Dalton Jones, Matthew J Morse +3
We demonstrate that geometrically distinctive keys during LLM inference tend to have high attention scores. Based on the phenomenon we propose KeyDiff, a training-free KV cache evi…
CAOTE: KV Cache Selection for LLMs via Attention Output Error-Based Token Eviction
Raghavv Goel, Junyoung Park, Mukul Gagrani +5
While long context support of large language models has extended their abilities, it also incurs challenges in memory and compute which becomes crucial bottlenecks in resource-rest…
BreezyVoice: Adapting TTS for Taiwanese Mandarin with Enhanced Polyphone Disambiguation -- Challenges and Insights
Chan-Jan Hsu, Yi-Cheng Lin, Chia-Chun Lin +10
We present BreezyVoice, a Text-to-Speech (TTS) system specifically adapted for Taiwanese Mandarin, highlighting phonetic control abilities to address the unique challenges of polyp…