2 papers
cs.LG2025
Paged Attention Meets FlexAttention: Unlocking Long-Context Efficiency in Deployed Inference
Thomas Joshi, Herman Saini, Neil Dhillon +2
Large Language Models (LLMs) encounter severe memory inefficiencies during long-context inference due to conventional handling of key-value (KV) caches. In this work, we introduce…
cs.CL2024
Enhancing Knowledge Distillation for LLMs with Response-Priming Prompting
Vijay Goyal, Mustafa Khan, Aprameya Tirupati +3
Large language models (LLMs) have demonstrated remarkable performance across a wide range of natural language processing (NLP) tasks. However, these models are often difficult to d…