5 citations · 9 across the 13 of their papers we have counts for
5 papers · 1 filter
Fast LLM Post-training via Decoupled and Fastest-of-N Speculation
Rongxin Cheng, Kai Zhou, Xingda Wei +8
Rollout dominates the training time in large language model (LLM) post-training, where the trained model is used to generate tokens given a batch of prompts. This work, SpecActor,…
Towards Lock Modularization for Heterogeneous Environments
Hanze Zhang, Rong Chen, Haibo Chen
Modern hardware environments are becoming increasingly heterogeneous, leading to the emergence of applications specifically designed to exploit this heterogeneity. Efficiently adop…
KVCache Cache in the Wild: Characterizing and Optimizing KVCache Cache at a Large Cloud Provider
Jiahao Wang, Jinbo Han, Xingda Wei +6
Serving large language models (LLMs) is important for cloud providers, and caching intermediate results (KV$) after processing each request substantially improves serving throughp…
DiFache: Efficient and Scalable Caching on Disaggregated Memory using Decentralized Coherence
Hanze Zhang, Kaiming Wang, Rong Chen +2
The disaggregated memory (DM) architecture offers high resource elasticity at the cost of data access performance. While caching frequently accessed data in compute nodes (CNs) red…
DecLock: A Case of Decoupled Locking for Disaggregated Memory
Hanze Zhang, Ke Cheng, Rong Chen +2
This paper reveals that locking can significantly degrade the performance of applications on disaggregated memory (DM), sometimes by several orders of magnitude, due to contention…