Full-Stack FP4: Stable LLM Pretraining with Quantized Projections, Optimizers, and Attention

Siyu Ding, Mingchuan Ma, Jiabo Tong +3

Recent NVFP4 pretraining work has primarily optimized Transformer linear projections, leaving persistent optimizer states, optimizer computation, and low-precision attention forwar…