SQ-format: A Unified Sparse-Quantized Hardware-friendly Data Format for LLMs

Ruixuan Huang, Hao Zeng, Hantao Huang +4

Post-training quantization (PTQ) plays a crucial role in the democratization of large language models (LLMs). However, existing low-bit quantization and sparsification techniques a…