4 papers
veScale-FSDP: Flexible and High-Performance FSDP at Scale
Zezhou Wang, Youjie Li, Zhiqi Lin +9
Fully Sharded Data Parallel (FSDP), also known as Zero Redundancy Optimizer (ZeRO), is widely used for large-scale model training, because of its memory efficiency and minimal intr…
Train Large, Deploy Compact: Structured Compression for Compact Low-Rank Adaptation
Xin Yu, Cong Xie, Ziyu Zhao +4
Low-rank adaptation (LoRA) has become a widely used paradigm for parameter-efficient fine-tuning of large language models, yet its representational capacity often lags behind full…
VeOmni: Scaling Any Modality Model Training with Model-Centric Distributed Recipe Zoo
Qianli Ma, Yaowei Zheng, Zhelun Shi +9
Recent advances in large language models (LLMs) have driven impressive progress in omni-modal understanding and generation. However, training omni-modal LLMs remains a significant…
Distributed Sign Momentum with Local Steps for Training Transformers
Shuhua Yu, Ding Zhou, Cong Xie +4
Pre-training Transformer models is resource-intensive, and recent studies have shown that sign momentum is an efficient technique for training large-scale deep learning models, par…