StagQ: Constraint-Driven Multi-Precision Weight Quantization for LLMs

Zhe Wei, Mengqi Guo, Yuan Yuan +3

Serving a large language model (LLM) across a fleet of deployments requires several weight-precision operating points. Multi-precision formats serve them all from one stream whose…