Greedy Local Learning for Language Model Pretraining: Gaps and Objective Design

Jihwan Moon, Sheir A. Zaheer, Jinmyoung Lee +2

Greedy block-wise local learning splits a network into gradient-isolated blocks trained by local auxiliary losses, deleting the backward pass between blocks: inter-stage communicat…