5 papers
Inference First, Training Underneath: Layer-Granular LoRA Co-Serving with LayerLoom
Zongpu Zhang, Yishu Yin, Xuehai Qian
Serving and fine-tuning a large language model on the same GPUs can turn the idle capacity between inference bursts into training progress, and low-rank adaptation (LoRA) makes thi…
GreedySnake: Accelerating SSD-Offloaded LLM Training with Efficient Scheduling and Optimizer Step Overlapping
Yishu Yin, Xuehai Qian
SSD-offloaded training offers a practical and promising approach to making LLM training cost-effective. Building on gradient accumulation with micro-batches, this paper introduces…
SRT: Accelerating Reinforcement Learning via Speculative Rollout with Tree-Structured Cache
Chi-Chih Chang, Siqi Zhu, Zhichen Zeng +5
We present Speculative Rollout with Tree-Structured Cache (SRT), a simple, model-free approach to accelerate on-policy reinforcement learning (RL) for language models without sacri…
Mesh-Attention: A New Communication-Efficient Distributed Attention with Improved Data Locality
Sirui Chen, Jingji Chen, Siqi Zhu +3
Distributed attention is essential for scaling large language models (LLMs) to long contexts, yet existing methods either have limited parallelism or incur high communication costs…
Fine-Grained Embedding Dimension Optimization During Training for Recommender Systems
Qinyi Luo, Penghan Wang, Wei Zhang +8
Huge embedding tables in modern deep learning recommender models (DLRM) require prohibitively large memory during training and inference. This paper proposes FIITED, a system to au…