176 citations · 826 across the 63 of their papers we have counts for
8 papers · 1 filter
Improving Generalization of Pre-trained Language Models via Stochastic Weight Averaging
Peng Lu, Ivan Kobyzev, Mehdi Rezagholizadeh +3
Knowledge Distillation (KD) is a commonly used technique for improving the generalization of compact Pre-trained Language Models (PLMs) on downstream tasks. However, such methods i…
Continuation KD: Improved Knowledge Distillation through the Lens of Continuation Optimization
Aref Jafari, Ivan Kobyzev, Mehdi Rezagholizadeh +2
Knowledge Distillation (KD) has been extensively used for natural language understanding (NLU) tasks to improve a small model's (a student) generalization by transferring the knowl…
DyLoRA: Parameter Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptation
Mojtaba Valipour, Mehdi Rezagholizadeh, Ivan Kobyzev +1
With the ever-growing size of pretrained models (PMs), fine-tuning them has become more expensive and resource-hungry. As a remedy, low-rank adapters (LoRA) keep the main pretraine…
The Sky Above The Clouds
Sarah Chasins, Alvin Cheung, Natacha Crooks +14
Technology ecosystems often undergo significant transformations as they mature. For example, telephony, the Internet, and PCs all started with a single provider, but in the United…
Do we need Label Regularization to Fine-tune Pre-trained Language Models?
Ivan Kobyzev, Aref Jafari, Mehdi Rezagholizadeh +5
Knowledge Distillation (KD) is a prominent neural model compression technique that heavily relies on teacher network predictions to guide the training of a student model. Consideri…
When Chosen Wisely, More Data Is What You Need: A Universal Sample-Efficient Strategy For Data Augmentation
Ehsan Kamalloo, Mehdi Rezagholizadeh, Ali Ghodsi
Data Augmentation (DA) is known to improve the generalizability of deep neural networks. Most existing DA techniques naively add a certain number of augmented samples without consi…