When Scaling Fails: Network and Fabric Effects on Distributed GPU Training Performance

Dinesh Gopalan, Ratul Ali

Scaling distributed GPU training is commonly assumed to yield predictable performance gains as additional nodes are added. In practice, many large-scale deployments encounter dimin…