3 papers
cs.LG2025
Towards Higher Effective Rank in Parameter-efficient Fine-tuning using Khatri--Rao Product
Paul Albert, Frederic Z. Zhang, Hemanth Saratchandran +2
Parameter-efficient fine-tuning (PEFT) has become a standard approach for adapting large pre-trained models. Amongst PEFT methods, low-rank adaptation (LoRA) has achieved notable s…
cs.CL2025
RandLoRA: Full-rank parameter-efficient fine-tuning of large models
Paul Albert, Frederic Z. Zhang, Hemanth Saratchandran +3
Low-Rank Adaptation (LoRA) and its variants have shown impressive results in reducing the number of trainable parameters and memory requirements of large transformer networks while…
cs.LG2024
Knowledge Composition using Task Vectors with Learned Anisotropic Scaling
Frederic Z. Zhang, Paul Albert, Cristian Rodriguez-Opazo +2
Pre-trained models produce strong generic representations that can be adapted via fine-tuning. The learned weight difference relative to the pre-trained model, known as a task vect…