A Distributed View from a Single GPU: RankLens for Distributed Training

Yufan Wang, Yihao Zhang, Yi Liu

Modern distributed training is costly to debug, tune, and analyze because faithful analysis often requires repeated target-scale executions. We observe that many analysis tasks dep…