2 citations · 2 across the 7 of their papers we have counts for
7 papers · 1 filter
Fast LLM Post-training via Decoupled and Fastest-of-N Speculation
Rongxin Cheng, Kai Zhou, Xingda Wei +8
Rollout dominates the training time in large language model (LLM) post-training, where the trained model is used to generate tokens given a batch of prompts. This work, SpecActor,…
Towards Lock Modularization for Heterogeneous Environments
Hanze Zhang, Rong Chen, Haibo Chen
Modern hardware environments are becoming increasingly heterogeneous, leading to the emergence of applications specifically designed to exploit this heterogeneity. Efficiently adop…
DiFache: Efficient and Scalable Caching on Disaggregated Memory using Decentralized Coherence
Hanze Zhang, Kaiming Wang, Rong Chen +2
The disaggregated memory (DM) architecture offers high resource elasticity at the cost of data access performance. While caching frequently accessed data in compute nodes (CNs) red…
DecLock: A Case of Decoupled Locking for Disaggregated Memory
Hanze Zhang, Ke Cheng, Rong Chen +2
This paper reveals that locking can significantly degrade the performance of applications on disaggregated memory (DM), sometimes by several orders of magnitude, due to contention…
KunServe: Parameter-centric Memory Management for Efficient Memory Overloading Handling in LLM Serving
Rongxin Cheng, Yuxin Lai, Xingda Wei +2
Serving LLMs with a cluster of GPUs is common nowadays, where the serving system must meet strict latency SLOs required by applications. However, the stateful nature of LLM serving…
BLITZSCALE: Fast and Live Large Model Autoscaling with O(1) Host Caching
Dingyan Zhang, Haotian Wang, Yang Liu +4
Model autoscaling is the key mechanism to achieve serverless model-as-a-service, but it faces a fundamental trade-off between scaling speed and storage/memory usage to cache parame…