4 papers
OjaKV: Context-Aware Online Low-Rank KV Cache Compression
Yuxuan Zhu, David H. Yang, Mohammad Mohammadi Amiri +3
The expanding long-context capabilities of large language models are constrained by a significant memory bottleneck: the key-value (KV) cache required for autoregressive generation…
ZoomR: Memory Efficient Reasoning through Multi-Granularity Key Value Retrieval
David H. Yang, Yuxuan Zhu, Mohammad Mohammadi Amiri +4
Large language models (LLMs) have shown great performance on complex reasoning tasks but often require generating long intermediate thoughts before reaching a final answer. During…
SentenceKV: Efficient LLM Inference via Sentence-Level Semantic KV Caching
Yuxuan Zhu, Ali Falahati, David H. Yang +1
Large language models face significant computational and memory challenges when processing long contexts. During inference, efficient management of the key-value (KV) cache, which…
Sparse Gradient Compression for Fine-Tuning Large Language Models
David H. Yang, Mohammad Mohammadi Amiri, Tejaswini Pedapati +2
Fine-tuning large language models (LLMs) for downstream tasks has become increasingly crucial due to their widespread use and the growing availability of open-source models. Howeve…