From the 1 of 4 linked papers with an AI index.
4 papers
SmartSwap: Swap-Based Memory Optimization for LLM Training under Varying Operator Sequences
Zibo Wang, Yuhang Zhou, Zhibin Wang +13
The paper introduces SmartSwap, a swap‑based memory optimization framework that adapts to varying operator sequences in eager‑mode training of large language models, using a lightw…
STAR: Decode-Phase Rescheduling for LLM Inference
Zhibin Wang, Zetao Hong, Xue Li +8
Large Language Model (LLM) inference has emerged as a fundamental paradigm, however, variations in output length cause severe workload imbalance in the decode phase, particularly f…
Accelerating Mixture-of-Experts Inference by Hiding Offloading Latency with Speculative Decoding
Zhibin Wang, Zhonghui Zhang, Yuhang Zhou +8
Recent advancements in Mixture of Experts (MoE) models have significantly increased their parameter scale as well as model performance. Extensive offloading techniques have been pr…
Echo: Efficient Co-Scheduling of Hybrid Online-Offline Tasks for Large Language Model Serving
Zhibin Wang, Shipeng Li, Xue Li +7
Large language models have been widely deployed in various applications, encompassing both interactive online tasks and batched offline tasks. Given the burstiness and latency sens…