On the Spectral Flattening of Quantized Embeddings

Junlin Huang, Wenyi Fang, Zhenheng Tang +5

Training Large Language Models (LLMs) at ultra-low precision is critically impeded by instability rooted in the conflict between discrete quantization constraints and the intrinsic…