Distributed Training under Packet Loss

Erez Weintraub, Ron Banner, Ariel Orda

State-of-the-art language and vision models are routinely trained across thousands of GPUs, often spanning multiple data-centers, yet today's distributed frameworks still assume re…