Distributed Low-Communication Training with Decoupled Momentum Optimization

Sasho Nedelkoski, Alexander Acker, Odej Kao +2

The training of large models demands substantial computational resources, typically available only in data centers with high-bandwidth interconnects. However, reducing the reliance…