3 papers
cs.DB2026
HierarchicalKV: A GPU Hash Table with Cache Semantics for Continuous Online Embedding Storage
Haidong Rong, Jiashu Yao, Matthias Langer +12
Traditional GPU hash tables preserve every inserted key -- a dictionary assumption that wastes scarce High Bandwidth Memory (HBM) when embedding tables routinely exceed single-GPU…
cs.LG2025
LiteCache: A Query Similarity-Driven, GPU-Centric KVCache Subsystem for Efficient LLM Inference
Jiawei Yi, Ping Gong, Youhui Bai +10
During LLM inference, KVCache memory usage grows linearly with sequence length and batch size and often exceeds GPU capacity. Recent proposals offload KV states to host memory and…
cs.DC2024
Hiding Communication Cost in Distributed LLM Training via Micro-batch Co-execution
Haiquan Wang, Chaoyi Ruan, Jia He +4
The growth of Large Language Models (LLMs) has necessitated large-scale distributed training. Highly optimized frameworks, however, still suffer significant losses in Model FLOPS u…