144 citations · 149 across the 13 of their papers we have counts for
13 papers
MosaicKV: Serving Long-Context LLM with Dynamic Two-D KV Cache Compression
Sheng Qiang, Ruiwei Chen, Yinpeng Wu +5
Long-context LLM services now sustain prompts with hundreds of thousands to millions of tokens, making the key-value (KV) cache a first-order serving cost. Because the cache grows…
DeltaBox: Scaling Stateful AI Agents with Millisecond-Level Sandbox Checkpoint/Rollback
Yunpeng Dong, Jingkai He, Shiqi Liu +7
LLM-powered AI agents require high-frequency state exploration (e.g., test-time tree search and reinforcement learning), relying on rapid checkpoint and rollback (C/R) of the compl…
Sharpen the Spec, Cut the Code: A Case for Generative File System with SYSSPEC
Qingyuan Liu, Mo Zou, Hengbin Zhang +3
File systems are critical OS components that require constant evolution to support new hardware and emerging application needs. However, the traditional paradigm of developing feat…
History Rhymes: Accelerating LLM Reinforcement Learning with RhymeRL
Jingkai He, Tianjian Li, Erhu Feng +5
With the rapid advancement of large language models (LLMs), reinforcement learning (RL) has emerged as a pivotal methodology for enhancing the reasoning capabilities of LLMs. Unlik…
Topology-Aware Virtualization over Inter-Core Connected Neural Processing Units
Dahu Feng, Erhu Feng, Dong Du +4
With the rapid development of artificial intelligence (AI) applications, an emerging class of AI accelerators, termed Inter-core Connected Neural Processing Units (NPU), has been a…
CHIME: A Case for Efficient Long-Context Attention-FC Disaggregated Inference with DIMM-PIM
Qingyuan Liu, Liyan Chen, Haocheng Wang +6
Attention-FC Disaggregated (AFD) LLM inference systems offload memory-bound Attention operations to memory-rich accelerators (e.g., CPUs, HBM-PIM) while retaining compute-bound Ful…
DIP: Efficient Large Multimodal Model Training with Dynamic Interleaved Pipeline
Zhenliang Xue, Hanpeng Hu, Xing Chen +7
Large multimodal models (LMMs) have demonstrated excellent capabilities in both understanding and generation tasks with various modalities. While these models can accept flexible c…
HeteroPod: XPU-Accelerated Infrastructure Offloading for Commodity Cloud-Native Applications
Bicheng Yang, Jingkai He, Dong Du +2
Cloud-native systems increasingly rely on infrastructure services (e.g., service meshes, monitoring agents), which compete for resources with user applications, degrading performan…
Characterizing Mobile SoC for Accelerating Heterogeneous LLM Inference
Le Chen, Dahu Feng, Erhu Feng +5
With the rapid advancement of artificial intelligence technologies such as ChatGPT, AI agents, and video generation, contemporary mobile systems have begun integrating these AI cap…
PowerInfer-2: Fast Large Language Model Inference on a Smartphone
Zhenliang Xue, Yixin Song, Zeyu Mi +3
Large language models (LLMs) on smartphones enable real-time AI assistance and privacy-preserving, offline operation. However, resource constraints of smartphones limit current dep…
Jiagu: Optimizing Serverless Computing Resource Utilization with Harmonized Efficiency and Practicability
Qingyuan Liu, Yanning Yang, Dong Du +5
Current serverless platforms struggle to optimize resource utilization due to their dynamic and fine-grained nature. Conventional techniques like overcommitment and autoscaling fal…
Occlum: Secure and Efficient Multitasking Inside a Single Enclave of Intel SGX
Youren Shen, Hongliang Tian, Yu Chen +4
Intel Software Guard Extensions (SGX) enables user-level code to create private memory regions called enclaves, whose code and data are protected by the CPU from software and hardw…
Taming Distrust in the Decentralized Internet with PIXIU
Yubin Xia, Qingyuan Liu, Cheng Tan +4
Decentralized Internet is booming. People are fascinated by its promise that users can truly own their data. However, in a decentralized Internet, completing a task usually involve…