3 papers
cs.AI2026
SnapStream: Efficient Long Sequence Decoding on Dataflow Accelerators
Jonathan Li, Nasim Farahini, Evgenii Iuliugin +19
The proliferation of 100B+ parameter Large Language Models (LLMs) with 100k+ context length support have resulted in increasing demands for on-chip memory to support large KV cache…
cs.AR2024
SambaNova SN40L: Scaling the AI Memory Wall with Dataflow and Composition of Experts
Raghu Prabhakar, Ram Sivaramakrishnan, Darshan Gandhi +27
Monolithic large language models (LLMs) like GPT-4 have paved the way for modern generative AI applications. Training, serving, and maintaining monolithic LLMs at scale, however, r…
cs.CL2024
Kernel Looping: Eliminating Synchronization Boundaries for Peak Inference Performance
David Koeplinger, Darshan Gandhi, Pushkar Nandkar +9
Token generation speed is critical to power the next wave of AI inference applications. GPUs significantly underperform during token generation due to synchronization overheads at…