AlignQuant: Tile-Aligned Mixed-Precision Quantization for Efficient LLM Generation

Hanzhi Zhang, Qiao Zhang, Qinglei Cao +4

Fine-grained mixed-precision quantization promises efficient large language model inference, but local precision choices can conflict with regular GPU storage and computation units…