7 citations · 22 across the 46 of their papers we have counts for
3 papers · 2 filters
Parameter-Efficient Fine-Tuning with Learnable Rank
Arpit Garg, Simon Lucey, Hemanth Saratchandran
Low-Rank Adaptation (LoRA) is a popular parameter-efficient fine-tuning (PEFT) method that restricts weight updates to low-rank adapters, introducing a fixed low-rank inductive bia…
Mask the Target: A Plug-and-Play Regularizer Against LoRA Forgetting
Runze Xu, Arpit Garg, Hemanth Saratchandran +1
Low-Rank Adaptation (LoRA) has become one of the most widely used fine-tuning mechanisms for adapting large language models to new domains, tasks, and users. Yet adaptation perform…
Procedural Pretraining: Warming Up Language Models with Abstract Data
Liangze Jiang, Zachary Shinnick, Anton van den Hengel +2
Pretraining language models directly on web-scale corpora is the de facto paradigm. We study an alternative where the model is initially exposed to abstract structured data to ease…