Gaussian Weight Sampling for Scalable, Efficient and Stable Pseudo-Quantization Training

Myeonghwan Ahn, Sungjoo Yoo

Ever-growing scale of large language models (LLMs) is pushing for improved efficiency, favoring fully quantized training (FQT) over BF16. While FQT accelerates training, it faces c…