4 papers
InstantInfer: Enabling Fast LLM Cold Start with Communicating Finite Automata
Yitao Yuan, Yongchao He, Shaoke Fang +1
Cold starts in large language model (LLM) inference services significantly affect user experience, yet they remain inefficient due to sequential initialization and a massive number…
Not All Tokens Are Worth Caching: Learning Semantic-Aware Eviction for LLM Prefix Caches
Shaoke Fang, Ziang Li, Wenfei Wu +3
Prefix caching is a key optimization in Large Language Model (LLM) serving, reusing attention Key-Value (KV) states across requests with shared prompt prefixes to reduce expensive…
EPIC: Abstraction and Polymorphism of In-Network Collectives on Ethernet
Yitao Yuan, Jianglong Nie, Tianyu Bai +28
In-Network Collective (INC) acceleration holds immense potential for optimizing AI training and inference; however, its cross-layer nature has historically hindered investment and…
HyperNAT: Scaling Up Network AddressTranslation with SmartNICs for Clouds
Shaoke Fang, Qingsong Liu, Wenfei Wu
Network address translation (NAT) is a basic functionality in cloud gateways. With the increasing traffic volume and number of flows introduced by the cloud tenants, the NAT gatewa…