RandLoRA: Full-rank parameter-efficient fine-tuning of large models

Paul Albert, Frederic Z. Zhang, Hemanth Saratchandran +3

Low-Rank Adaptation (LoRA) and its variants have shown impressive results in reducing the number of trainable parameters and memory requirements of large transformer networks while…