ResBM: Residual Bottleneck Models for Low-Bandwidth Pipeline Parallelism

Alan Aboudib, Rodrigo Lopez Portillo A., Kalei Brady +1

Unlocking large-scale low-bandwidth decentralized training has the potential to utilize otherwise untapped compute resources. In centralized settings, large-scale multi-node traini…