QuAILoRA: Quantization-Aware Initialization for LoRA

Neal Lawton, Aishwarya Padmakumar, Judith Gaspers +4

QLoRA reduces the memory-cost of fine-tuning a large language model (LLM) with LoRA by quantizing the base LLM. However, quantization introduces quantization errors that negatively…