5 papers
The Curse and Blessing of Mean Bias in FP4-Quantized LLM Training
Hengjie Cao, Zhendong Huang, Mengyi Chen +15
FP4 training promises substantial memory and compute savings for large language models, but remains fragile because blockwise quantization is dictated by extreme activation magnitu…
DecodeShare: Tracing the Shared Subspace of LLM Decode-Time Decisions
Zishan Shao, Lixun Zhang, Kangning Cui +10
Large language models (LLMs) handle many tasks with one set of parameters, but under KV-cached inference it is unclear what task-general structure, if any, is used at decode time r…
Multi-Stream Perturbation Attack: Breaking Safety Alignment of Thinking LLMs Through Concurrent Task Interference
Fan Yang
The widespread adoption of thinking mode in large language models (LLMs) has significantly enhanced complex task processing capabilities while introducing new security risks. When…
Spectra: Rethinking Optimizers for LLMs Under Spectral Anisotropy
Zhendong Huang, Hengjie Cao, Fang Dong +14
Gradient signals in LLM training are highly anisotropic: recurrent linguistic structure concentrates energy into a small set of dominant spectral directions, while context specific…
Metis: Training LLMs with FP4 Quantization
Hengjie Cao, Mengyi Chen, Yifeng Yang +13
This work identifies anisotropy in the singular value spectra of parameters, activations, and gradients as the fundamental barrier to low-bit training of large language models (LLM…