4 papers
CoRun: Padding is Simple and Efficient for Deterministic LLM Inference
Shiju Zhao, Jiacheng Yang, Qihang Chen +4
Despite fixed sampling parameters and random seeds, Large Language Model (LLM) inference exhibits output inconsistency, which undermines downstream tasks such as model evaluation a…
veScale-FSDP: Flexible and High-Performance FSDP at Scale
Zezhou Wang, Youjie Li, Zhiqi Lin +9
Fully Sharded Data Parallel (FSDP), also known as Zero Redundancy Optimizer (ZeRO), is widely used for large-scale model training, because of its memory efficiency and minimal intr…
veScale: Consistent and Efficient Tensor Programming with Eager-Mode SPMD
Youjie Li, Cheng Wan, Zhiqi Lin +10
Large Language Models (LLMs) have scaled rapidly in size and complexity, requiring increasingly intricate parallelism for distributed training, such as 3D parallelism. This sophist…
VeOmni: Scaling Any Modality Model Training with Model-Centric Distributed Recipe Zoo
Qianli Ma, Yaowei Zheng, Zhelun Shi +9
Recent advances in large language models (LLMs) have driven impressive progress in omni-modal understanding and generation. However, training omni-modal LLMs remains a significant…