3 papers
cs.LG2026
Format-Aware Fusion for Fast FP4 Pretraining
Robert Hu
Four-bit floating-point (FP4) Tensor Cores accelerate matrix multiplication, but scale computation, operand packing, layout construction, and saved backward state can erase the gai…
cs.LG2026
UE5M3 FP4 Block Scaling for Stable Language Model Pretraining
Robert Hu, Carlo Luschi, Paul Balanca
Stable 4-bit floating-point (FP4) pretraining is difficult because the E2M1 payload represents only a narrow range of magnitudes. NVIDIA's Transformer Engine \nv{} recipe addresses…
cs.LG2025
Elucidating the Design Space of FP4 training
Robert Hu, Carlo Luschi, Paul Balanca
The increasing computational demands of foundation models have spurred research into low-precision training, with 4-bit floating-point (\texttt{FP4}) formats emerging as a frontier…