PC-LoRA: Low-Rank Adaptation for Progressive Model Compression with Knowledge Distillation

Injoon Hwang, Haewon Park, Youngwan Lee +2

Low-rank adaption (LoRA) is a prominent method that adds a small number of learnable parameters to the frozen pre-trained weights for parameter-efficient fine-tuning. Prompted by t…