GEM-Style Constraints for PEFT with Dual Gradient Projection in LoRA

Brian Tekmen, Jason Yin, Qianqian Tong

Full fine-tuning of Large Language Models (LLMs) is computationally costly, motivating Continual Learning (CL) approaches that utilize parameter-efficient adapters. We revisit Grad…