2 papers
cs.AI2025
SnapStream: Efficient Long Sequence Decoding on Dataflow Accelerators
Jonathan Li, Nasim Farahini, Evgenii Iuliugin +19
The proliferation of 100B+ parameter Large Language Models (LLMs) with 100k+ context length support have resulted in increasing demands for on-chip memory to support large KV cache…
cs.CL2024
Kernel Looping: Eliminating Synchronization Boundaries for Peak Inference Performance
David Koeplinger, Darshan Gandhi, Pushkar Nandkar +9
Token generation speed is critical to power the next wave of AI inference applications. GPUs significantly underperform during token generation due to synchronization overheads at…