FCDP: Fully Cached Data Parallel for Communication-Avoiding Large-Scale Training

Gyeongseo Park, Eungyeong Lee, Song-woo Sok +5

Training billion-parameter models requires distributing model states across GPUs using fully sharded data parallel (i.e., ZeRO-3). While ZeRO-3 succeeds on clusters with high-bandw…