Metis: Training LLMs with FP4 Quantization

Hengjie Cao, Mengyi Chen, Yifeng Yang +13

This work identifies anisotropy in the singular value spectra of parameters, activations, and gradients as the fundamental barrier to low-bit training of large language models (LLM…