QLoRA: Efficient Finetuning of Quantized LLMs

Abstract

We present QLoRA, an efficient finetuning approach that reduces memory usage enough to finetune a 65B parameter model on a single 48GB GPU while preserving full 16-bit finetuning task performance. QLoRA backpropagates gradients through a frozen, 4-bit quantized pretrained language model into Low Rank Adapters~(LoRA). Our best model family, which we name Guanaco, outperforms all previous openly released models on the Vicuna benchmark, reaching 99.3% of the performance level of ChatGPT while only requiring 24 hours of finetuning on a single GPU. QLoRA introduces a number of innovations to save memory without sacrificing performance: (a) 4-bit NormalFloat (NF4), a new data type that is information theoretically optimal for normally distributed weights (b) double quantization to reduce the average memory footprint by quantizing the quantization constants, and (c) paged optimziers to manage memory spikes. We use QLoRA to finetune more than 1,000 models, providing a detailed analysis of instruction following and chatbot performance across 8 instruction datasets, multiple model types (LLaMA, T5), and model scales that would be infeasible to run with regular finetuning (e.g. 33B and 65B parameter models). Our results show that QLoRA finetuning on a small high-quality dataset leads to state-of-the-art results, even when using smaller models than the previous SoTA. We provide a detailed analysis of chatbot performance based on both human and GPT-4 evaluations showing that GPT-4 evaluations are a cheap and reasonable alternative to human evaluation. Furthermore, we find that current chatbot benchmarks are not trustworthy to accurately evaluate the performance levels of chatbots. A lemon-picked analysis demonstrates where Guanaco fails compared to ChatGPT. We release all of our models and code, including CUDA kernels for 4-bit training.

arXiv commentExtended NeurIPS submission

Cited by in corpus (27)

• Adapted Large Language Models Can Outperform Medical Experts in Clinical Text Summarization
• ChatEDA: A Large Language Model Powered Autonomous Agent for EDA
• Enhancing Phenotype Recognition in Clinical Notes Using Large Language Models: PhenoBCBERT and PhenoGPT
• Fine-tuning and Utilization Methods of Domain-specific LLMs
• Exploring the Roles of Large Language Models in Reshaping Transportation Systems: A Survey, Framework, and Roadmap
• Enhancing textual textbook question answering with large language models and retrieval augmented generation
• From Screens to Scenes: A Survey of Embodied AI in Healthcare
• A dataset and benchmark for hospital course summarization with adapted large language models
• Playing with words: Comparing the vocabulary and lexical diversity of ChatGPT and humans
• Domain-specific ChatBots for Science using Embeddings
• Exploring the Potential of Large Language Models for Improving Digital Forensic Investigation Efficiency
• Harnessing Earnings Reports for Stock Predictions: A QLoRA-Enhanced LLM Approach
• Exploring Large Language Models to generate Easy to Read content
• Zero-shot Bilingual App Reviews Mining with Large Language Models
• CARE-SD: Classifier-based analysis for recognizing and eliminating stigmatizing and doubt marker labels in electronic health records: model development and validation
• Foundational Models and Federated Learning: Survey, Taxonomy, Challenges and Practical Insights
• Detect Llama -- Finding Vulnerabilities in Smart Contracts using Large Language Models
• Democratizing LLMs: An Exploration of Cost-Performance Trade-offs in Self-Refined Open-Source Models
• LLM-based speaker diarization correction: A generalizable approach
• Deep Learning and LLM-based Methods Applied to Stellar Lightcurve Classification
• LEGO-GraphRAG: Modularizing Graph-based Retrieval-Augmented Generation for Design Space Exploration
• Free and Customizable Code Documentation with LLMs: A Fine-Tuning Approach
• Evaluating Language Model Agency through Negotiations
• Speaker attribution in German parliamentary debates with QLoRA-adapted large language models
• Camera Control at the Edge with Language Models for Scene Understanding
• The Complexity Dynamics of Grokking
• ModuLoRA: Finetuning 2-Bit LLMs on Consumer GPUs by Integrating with Modular Quantizers