8-bit Optimizers via Block-wise Quantization

Tim Dettmers, Mike Lewis, Sam Shleifer +1

Stateful optimizers maintain gradient statistics over time, e.g., the exponentially smoothed sum (SGD with momentum) or squared sum (Adam) of past gradient values. This state can b…