activity
20222026
most citedSpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification

101 citations · 162 across the 6 of their papers we have counts for

collaborators
Showing 2026Show all

1 paper · 1 filter

cs.AR2026

Multi-Segment Attention: Enabling Efficient KV-Cache Management for Faster Large Language Model Serving

Chunan Shi, Yilei Chen, Yilin Chen +2

Large Language Model (LLM) inference relies on key-value (KV) caches to avoid redundant attention computation. While approximate KV cache retention techniques reduce memory usage b…