ModuLoRA: Finetuning 2-Bit LLMs on Consumer GPUs by Integrating with Modular Quantizers

Abstract

We propose a memory-efficient finetuning algorithm for large language models (LLMs) that supports finetuning LLMs with 65B parameters in 2/3/4-bit precision on as little as one 24GB GPU. Our method, modular low-rank adaptation (ModuLoRA), integrates any user-specified weight quantizer with finetuning via low-rank adapters (LoRAs). Our approach relies on a simple quantization-agnostic backward pass that adaptively materializes low-precision LLM weights from a custom black-box quantization module. This approach enables finetuning 2-bit and 3-bit LLMs for the first time -- leveraging state-of-the-art 2-bit QuIP\# quantization and 3-bit OPTQ quantization -- outperforming finetuning that relies on less sophisticated 4-bit and 8-bit methods. In our experiments, \lplora~attains competitive performance on text classification, natural language inference, and instruction following tasks using significantly less memory than existing approaches, and we also surpass the state-of-the-art ROUGE score on a popular summarization task. We release \lplora~together with a series of low-precision models as part of \llmtune, a user-friendly library for quantizing, running, and finetuning LLMs on consumer GPUs.

arXiv commentUpdate since being accepted to TMLR. Updated 2Bit results

References in corpus (13)

• LLaMA: Open and Efficient Foundation Language Models
• Scaling Instruction-Finetuned Language Models
• QLoRA: Efficient Finetuning of Quantized LLMs
• Learning How to Ask: Querying LMs with Mixtures of Soft Prompts
• Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning
• LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale
• SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
• ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers
• INSTRUCTEVAL: Towards Holistic Evaluation of Instruction-Tuned Large Language Models
• QuIP: 2-Bit Quantization of Large Language Models With Guarantees
• RPTQ: Reorder-based Post-training Quantization for Large Language Models
• LUT-GEMM: Quantized Matrix Multiplication based on LUTs for Efficient Inference in Large-Scale Generative Language Models
• Input-Tuning: Adapting Unfamiliar Inputs to Frozen Pretrained Models