Accelerating Sharded Data Parallelism at Scale with Federated Learning

Gianluca Mittone, Marco Aldinucci

The symbiotic scaling of artificial intelligence models and high-performance computing systems continually creates algorithmic challenges in their convergence. Foundation models (F…