Simulating Unified Tensor Resharding in heterogeneous AI systems

Sumit Kumar, Sayantan Dasgupta, Kushal Mitra +6

State-of-the-art AI training simulators assume homogeneous compute and network infrastructure. However, real-world training infrastructure is becoming increasingly heterogeneous si…