Breaking the Bubble: Asynchronous Pipeline Parallel Training with Bounded Weight Inconsistency

Itay Elam, Eliron Rahimi, Avi Mendelson +1

Pipeline parallelism is essential for training large neural networks, but existing schedules trade off throughput, memory, and optimization consistency. Synchronous pipelines prese…