2 papers
cs.LG2026
COBS: Cumulant Order Block Sparse Attention
Alexander Tian, Aditya Ghai, Sanjit Neelam +2
Block sparse attention is a hardware friendly way to alleviate the key-value (KV) cache read bottleneck in large language models (LLMs). However, it is not prevalent among leading…
cs.LG2025
SPIRe: Boosting LLM Inference Throughput with Speculative Decoding
Sanjit Neelam, Daniel Heinlein, Vaclav Cvicek +2
Speculative decoding (SD) has been shown to reduce the latency of autoregressive decoding (AD) by 2-3x for small batch sizes. However, increasing throughput and therefore reducing…