11 papers
xKV: Cross-Layer KV-Cache Compression via Aligned Singular Vector Extraction
Chi-Chih Chang, Wei-Cheng Lin, Chien-Yu Lin +8
Long-context Large Language Models (LLMs) enable powerful applications but incur high memory costs due to the key-value states (KV-Cache). Recent studies attempt to share KV-Cache…
TokenButler: Token Importance is Predictable
Yash Akhauri, Ahmed F AbouElhamayed, Yifei Gao +4
Large Language Models (LLMs) rely on the Key-Value (KV) Cache to store token history, enabling efficient decoding of tokens. As the KV-Cache grows, it becomes a major memory and co…
DisagMoE: Computation-Communication overlapped MoE Training via Disaggregated AF-Pipe Parallelism
Zhichen Zeng, Chi-Chih Chang, Jiayi Wang +10
Mixture-of-experts (MoE) architectures enable trillion-parameter LLMs with sparsely activated experts. Expert parallelism (EP) is a widely adopted MoE training strategy, but it suf…
DARE: Diffusion Language Model Activation Reuse for Efficient Inference
Natalia Frumkin, Bokun Wang, Hung-Yueh Chiang +3
Diffusion Large Language Models (dLLMs) have emerged as a promising alternative to auto-regressive (AR) models, offering greater expressive capacity and potential for parallel gene…
Faster LLM Inference via Sequential Monte Carlo
Yahya Emara, Mauricio Barba da Costa, Chi-Chih Chang +4
Speculative decoding (SD) accelerates language model inference by drafting tokens from a cheap proposal model and verifying them against an expensive target model via rejection sam…
UniQL: Unified Quantization and Low-rank Compression for Adaptive Edge LLMs
Hung-Yueh Chiang, Chi-Chih Chang, Yu-Chen Lu +4
Deploying large language models (LLMs) on mobile platforms faces significant challenges due to the limited memory and shared computational resources of the device. Resource availab…