FineQuant: Unlocking Efficiency with Fine-Grained Weight-Only Quantization for LLMs

Young Jin Kim, Rawn Henry, Raffy Fahim +1

Large Language Models (LLMs) have achieved state-of-the-art performance across various language tasks but pose challenges for practical deployment due to their substantial memory r…