activity
20242026
most citedMegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs

24 citations · 26 across the 7 of their papers we have counts for

collaborators
Showing cs.DCShow all

1 paper · 1 filter

cs.DC2026

veScale-FSDP: Flexible and High-Performance FSDP at Scale

Zezhou Wang, Youjie Li, Zhiqi Lin +9

Fully Sharded Data Parallel (FSDP), also known as Zero Redundancy Optimizer (ZeRO), is widely used for large-scale model training, because of its memory efficiency and minimal intr…