8 papers
RefLoRA: Refactored Low-Rank Adaptation for Efficient Fine-Tuning of Large Models
Yilang Zhang, Bingcong Li, Georgios B. Giannakis
Low-Rank Adaptation (LoRA) lowers the computational and memory overhead of fine-tuning large models by updating a low-dimensional subspace of the pre-trained weight matrix. Albeit…
ScaLoRA: Optimally Scaled Low-Rank Adaptation for Efficient High-Rank Fine-Tuning
Yilang Zhang, Xiaodong Yang, Yiwei Cai +1
As large language models (LLMs) continue to scale in size, the computational overhead has become a major bottleneck for task-specific fine-tuning. While low-rank adaptation (LoRA)…
Low-Rank Adaptation Redux for Large Models
Bingcong Li, Yilang Zhang, Georgios B. Giannakis
Low-rank adaptation (LoRA) has emerged as the de facto standard for parameter-efficient fine-tuning (PEFT) of foundation models, enabling the adaptation of billion-parameter networ…
Binomial Gradient-Based Meta-Learning for Enhanced Meta-Gradient Estimation
Yilang Zhang, Abraham Jaeger Mountain, Bingcong Li +1
Meta-learning offers a principled framework leveraging \emph{task-invariant} priors from related tasks, with which \emph{task-specific} models can be fine-tuned on downstream tasks…
ANCRe: Adaptive Neural Connection Reassignment for Efficient Depth Scaling
Yilang Zhang, Bingcong Li, Niao He +1
Scaling network depth has been a central driver behind the success of modern foundation models, yet recent investigations suggest that deep layers are often underutilized. This pap…
VASSO: Variance Suppression for Sharpness-Aware Minimization
Bingcong Li, Yilang Zhang, Georgios B. Giannakis
Sharpness-aware minimization (SAM) has well-documented merits in enhancing generalization of deep neural network models. Accounting for sharpness in the loss function geometry, whe…