3 papers
cs.LG2026
LiteCache: A Query Similarity-Driven, GPU-Centric KVCache Subsystem for Efficient LLM Inference
Jiawei Yi, Ping Gong, Youhui Bai +10
During LLM inference, KVCache memory usage grows linearly with sequence length and batch size and often exceeds GPU capacity. Recent proposals offload KV states to host memory and…
cs.DB2026
HierarchicalKV: A GPU Hash Table with Cache Semantics for Continuous Online Embedding Storage
Haidong Rong, Jiashu Yao, Matthias Langer +12
Traditional GPU hash tables preserve every inserted key -- a dictionary assumption that wastes scarce High Bandwidth Memory (HBM) when embedding tables routinely exceed single-GPU…
cs.DC2024
Hiding Communication Cost in Distributed LLM Training via Micro-batch Co-execution
Haiquan Wang, Chaoyi Ruan, Jia He +4
The growth of Large Language Models (LLMs) has necessitated large-scale distributed training. Highly optimized frameworks, however, still suffer significant losses in Model FLOPS u…