17 citations · 19 across the 17 of their papers we have counts for
17 papers
vToken: Token-Level Virtualization for Reclaimable KV Caches
Yuanhang Gao, Xiangrui Yang, Yuanfeng Chen +4
Large language model serving faces a critical memory bottleneck: the KV cache grows with sequence length and batch size. PagedAttention uses fixed-size memory blocks to reduce allo…
InstantInfer: Enabling Fast LLM Cold Start with Communicating Finite Automata
Yitao Yuan, Yongchao He, Shaoke Fang +1
Cold starts in large language model (LLM) inference services significantly affect user experience, yet they remain inefficient due to sequential initialization and a massive number…
Not All Tokens Are Worth Caching: Learning Semantic-Aware Eviction for LLM Prefix Caches
Shaoke Fang, Ziang Li, Wenfei Wu +3
Prefix caching is a key optimization in Large Language Model (LLM) serving, reusing attention Key-Value (KV) states across requests with shared prompt prefixes to reduce expensive…
EPIC: Abstraction and Polymorphism of In-Network Collectives on Ethernet
Yitao Yuan, Jianglong Nie, Tianyu Bai +28
In-Network Collective (INC) acceleration holds immense potential for optimizing AI training and inference; however, its cross-layer nature has historically hindered investment and…
LCMP: Distributed Long-Haul Cost-Aware Multi-Path Routing for Inter-Datacenter RDMA Networks
Dong-Yang Yu, Yuchao Zhang, Xiaodi Wang +5
RDMA-empowered cloud services are gradually deployed across datacenters (DCs) with multiple paths, which exhibit new properties of path asymmetry, delayed congestion signals, and s…
VeriNC: Finding Design Risks of In-Network Computing Systems
Tianyu Bai, Xiaoxi Zhang, Haoqing Wang +2
The emergence of programmable switches has brought in-network computing (INC) into the spotlight in recent years. By offloading computation directly onto the data transmission proc…
M2K: Making the Model-Kernel Interface Explicit for Reliable CUDA Kernel Verification
Mengting He, Shihao Xia, Haomin Jia +2
Large language model (LLM) inference systems rely on CUDA kernels for core GPU computations, yet the interface between models and kernels is implicit and poorly specified. Models a…
Fine-grained MoE Load Balancing with Linear Programming
Chenqi Zhao, Wenfei Wu, Linhai Song +2
Mixture-of-Experts (MoE) has emerged as a promising approach to scale up deep learning models due to its significant reduction in computational resources. However, the dynamic natu…
XBOF: A Cost-Efficient CXL JBOF with Inter-SSD Compute Resource Sharing
Shushu Yi, Yuda An, Li Peng +11
Enterprise SSDs integrate numerous computing resources (e.g., ARM processor and onboard DRAM) to satisfy the ever-increasing performance requirements of I/O bursts. While these res…
Selective KV-Cache Sharing to Mitigate Timing Side-Channels in LLM Inference
Kexin Chu, Zecheng Lin, Dawei Xiang +7
Global KV-cache sharing is an effective optimization for accelerating large language model (LLM) inference, yet it introduces an API-visible timing side channel that lets adversari…
AllReduce Scheduling with Hierarchical Deep Reinforcement Learning
Yufan Wei, Mickel Liu, Wenfei Wu
AllReduce is a technique in distributed computing which saw use in many critical applications of deep learning. Existing methods of AllReduce scheduling oftentimes lack flexibility…
Accelerating Distributed Deep Learning using Lossless Homomorphic Compression
Haoyu Li, Yuchen Xu, Jiayi Chen +5
As deep neural networks (DNNs) grow in complexity and size, the resultant increase in communication overhead during distributed training has become a significant bottleneck, challe…
ClickINC: In-network Computing as a Service in Heterogeneous Programmable Data-center Networks
Wenquan Xu, Zijian Zhang, Yong Feng +8
In-Network Computing (INC) has found many applications for performance boosts or cost reduction. However, given heterogeneous devices, diverse applications, and multi-path network…
NetRPC: Enabling In-Network Computation in Remote Procedure Calls
Bohan Zhao, Wenfei Wu, Wei Xu
People have shown that in-network computation (INC) significantly boosts performance in many application scenarios include distributed training, MapReduce, agreement, and network m…
Efficient Data-Plane Memory Scheduling for In-Network Aggregation
Hao Wang, Yuxuan Qin, ChonLam Lao +3
As the scale of distributed training grows, communication becomes a bottleneck. To accelerate the communication, recent works introduce In-Network Aggregation (INA), which moves th…
HyperNAT: Scaling Up Network AddressTranslation with SmartNICs for Clouds
Shaoke Fang, Qingsong Liu, Wenfei Wu
Network address translation (NAT) is a basic functionality in cloud gateways. With the increasing traffic volume and number of flows introduced by the cloud tenants, the NAT gatewa…
Simultaneously Achieving Sublinear Regret and Constraint Violations for Online Convex Optimization with Time-varying Constraints
Qingsong Liu, Wenfei Wu, Longbo Huang +1
In this paper, we develop a novel virtual-queue-based online algorithm for online convex optimization (OCO) problems with long-term and time-varying constraints and conduct a perfo…