MoBiQuant: Mixture-of-Bits Quantization for Token-Adaptive Any-Precision LLM

Dongwei Wang, Jinhee Kim, Seokho Han +8

Dynamic runtime latency and memory constraints necessitate flexible large language model (LLM) deployment, where an LLM can be inferred with various quantization precisions based o…