Stable FP4 Training via Transposition-Invariant Block Quantization

Mehdi Rahimifar, Amin Darabi, Mehran Taghian Jazi +6

Reducing training precision is a key lever for improving the e ciency of large language model (LLM) training, but pushing beyond FP8 to 4-bit oating point (FP4) remains challenging…