17 citations · 19 across the 17 of their papers we have counts for
7 papers · 1 filter
vToken: Token-Level Virtualization for Reclaimable KV Caches
Yuanhang Gao, Xiangrui Yang, Yuanfeng Chen +4
Large language model serving faces a critical memory bottleneck: the KV cache grows with sequence length and batch size. PagedAttention uses fixed-size memory blocks to reduce allo…
InstantInfer: Enabling Fast LLM Cold Start with Communicating Finite Automata
Yitao Yuan, Yongchao He, Shaoke Fang +1
Cold starts in large language model (LLM) inference services significantly affect user experience, yet they remain inefficient due to sequential initialization and a massive number…
Not All Tokens Are Worth Caching: Learning Semantic-Aware Eviction for LLM Prefix Caches
Shaoke Fang, Ziang Li, Wenfei Wu +3
Prefix caching is a key optimization in Large Language Model (LLM) serving, reusing attention Key-Value (KV) states across requests with shared prompt prefixes to reduce expensive…
EPIC: Abstraction and Polymorphism of In-Network Collectives on Ethernet
Yitao Yuan, Jianglong Nie, Tianyu Bai +28
In-Network Collective (INC) acceleration holds immense potential for optimizing AI training and inference; however, its cross-layer nature has historically hindered investment and…
LCMP: Distributed Long-Haul Cost-Aware Multi-Path Routing for Inter-Datacenter RDMA Networks
Dong-Yang Yu, Yuchao Zhang, Xiaodi Wang +5
RDMA-empowered cloud services are gradually deployed across datacenters (DCs) with multiple paths, which exhibit new properties of path asymmetry, delayed congestion signals, and s…
VeriNC: Finding Design Risks of In-Network Computing Systems
Tianyu Bai, Xiaoxi Zhang, Haoqing Wang +2
The emergence of programmable switches has brought in-network computing (INC) into the spotlight in recent years. By offloading computation directly onto the data transmission proc…
M2K: Making the Model-Kernel Interface Explicit for Reliable CUDA Kernel Verification
Mengting He, Shihao Xia, Haomin Jia +2
Large language model (LLM) inference systems rely on CUDA kernels for core GPU computations, yet the interface between models and kernels is implicit and poorly specified. Models a…