1.3k citations · 1.4k across the 26 of their papers we have counts for
3 papers · 2 filters
KronA: Parameter Efficient Tuning with Kronecker Adapter
Ali Edalati, Marzieh Tahaei, Ivan Kobyzev +3
Fine-tuning a Pre-trained Language Model (PLM) on a specific downstream task has been a well-known paradigm in Natural Language Processing. However, with the ever-growing size of P…
Improving Generalization of Pre-trained Language Models via Stochastic Weight Averaging
Peng Lu, Ivan Kobyzev, Mehdi Rezagholizadeh +3
Knowledge Distillation (KD) is a commonly used technique for improving the generalization of compact Pre-trained Language Models (PLMs) on downstream tasks. However, such methods i…
DyLoRA: Parameter Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptation
Mojtaba Valipour, Mehdi Rezagholizadeh, Ivan Kobyzev +1
With the ever-growing size of pretrained models (PMs), fine-tuning them has become more expensive and resource-hungry. As a remedy, low-rank adapters (LoRA) keep the main pretraine…