Hiding Communication Cost in Distributed LLM Training via Micro-batch Co-execution

Haiquan Wang, Chaoyi Ruan, Jia He +4

The growth of Large Language Models (LLMs) has necessitated large-scale distributed training. Highly optimized frameworks, however, still suffer significant losses in Model FLOPS u…