NestPipe: Large-Scale Recommendation Training on 1,500+ Accelerators via Nested Pipelining

Zhida Jiang, Zhaolong Xing, Huichao Chai +12

Modern recommendation models have increased to trillions of parameters. As cluster scales expand to O(1k), distributed training bottlenecks shift from computation and memory to dat…