Tied-Lora: Enhancing parameter efficiency of LoRA with weight tying

Adithya Renduchintala, Tugrul Konuk, Oleksii Kuchaiev

We introduce Tied-LoRA, a novel paradigm leveraging weight tying and selective training to enhance the parameter efficiency of Low-rank Adaptation (LoRA). Our exploration encompass…