Lagom: Unleashing the Power of Communication and Computation Overlapping for Distributed LLM Training

Guanbin Xu, ZhenGuo Xu, Yuzhe Li +4

Overlapping communication with computation is crucial for distributed large-model training, yet optimizing it - especially when computation becomes the bottleneck-remains challengi…