Compass: Dissecting Communication and Computation Operators for Efficient LLM Training

Guangyu Xiang, Lin Zhang, Haoxuan Yu +3

Overlapping communication and computation operators is a common practice to hide communication overheads, accelerating large language models (LLMs) training on GPU clusters. Existi…