6 papers
SARIA: Persistence-Aware KV Caching with Coordinated DRAM--SSD Reload for LLM Agents
Hui Li, Qihui Zhou, Xiao Yan +7
Multi-turn LLM agents usually reuse long context prefixes, making key value (KV) caching crucial for reducing prefill latency. SSDs provide a low-cost extension to limited DRAM cap…
QUILT: Rethinking Sparse-Attention Prefill through Shared Query Execution
Zhenduo Zhao, Qihui Zhou, Mingcong Song +7
Sparse attention reduces the cost of long-context attention, but existing kernels typically process queries independently, repeatedly loading and dequantizing KV entries shared acr…
From Experience to Expertise: Adoption-Aware Memory Learning for Data-Scarce NPU Kernel Synthesis
Longxiao Fan, Tao Zhang, Han Yan +5
High-performance kernels underpin efficient accelerator execution but require expert tuning and lengthy manual optimization cycles. LLM coding agents promise automation, yet their…
Metis: Bridging Text and Code Memory for Self-Evolving Agents
Zijie Dai, Siuhin He, Hui Li +9
Self-evolving agents improve over time by distilling experience from past executions and reusing it in future tasks. Existing systems represent such experience either as natural-la…
LoPA: Scaling dLLM Inference via Lookahead Parallel Decoding
Chenkai Xu, Yijie Jin, Jiajun Li +8
Diffusion Large Language Models (dLLMs) have demonstrated significant potential for high-speed inference. However, current confidence-driven decoding strategies are constrained by…
Tackling the Dynamicity in a Production LLM Serving System with SOTA Optimizations via Hybrid Prefill/Decode/Verify Scheduling on Efficient Meta-kernels
Mingcong Song, Xinru Tang, Fengfan Hou +9
Meeting growing demands for low latency and cost efficiency in production-grade large language model (LLM) serving systems requires integrating advanced optimization techniques. Ho…