FORGE: Fused On-Register Gradient Elimination for Memory-Efficient LLM Training

Dikshant Kukreja, Kritarth Prasad, Avinash Anand +6

Reverse-mode differentiation computes every weight gradient, writes it to memory, and only then lets the optimizer read it back. This two-phase schedule sets the memory ceiling of…