3 papers
cs.CL2025
LLMs Know What to Drop: Self-Attention Guided KV Cache Eviction for Efficient Long-Context Inference
Guangtao Wang, Shubhangi Upasani, Chen Wu +5
Efficient long-context inference is critical as large language models (LLMs) adopt context windows of ranging from 128K to 1M tokens. However, the growing key-value (KV) cache and…
cs.AR2024
SambaNova SN40L: Scaling the AI Memory Wall with Dataflow and Composition of Experts
Raghu Prabhakar, Ram Sivaramakrishnan, Darshan Gandhi +27
Monolithic large language models (LLMs) like GPT-4 have paved the way for modern generative AI applications. Training, serving, and maintaining monolithic LLMs at scale, however, r…
cs.CL2024
Kernel Looping: Eliminating Synchronization Boundaries for Peak Inference Performance
David Koeplinger, Darshan Gandhi, Pushkar Nandkar +9
Token generation speed is critical to power the next wave of AI inference applications. GPUs significantly underperform during token generation due to synchronization overheads at…