9 papers
Low-Rank Adaptation Redux for Large Models
Bingcong Li, Yilang Zhang, Georgios B. Giannakis
Low-rank adaptation (LoRA) has emerged as the de facto standard for parameter-efficient fine-tuning (PEFT) of foundation models, enabling the adaptation of billion-parameter networ…
Binomial Gradient-Based Meta-Learning for Enhanced Meta-Gradient Estimation
Yilang Zhang, Abraham Jaeger Mountain, Bingcong Li +1
Meta-learning offers a principled framework leveraging \emph{task-invariant} priors from related tasks, with which \emph{task-specific} models can be fine-tuned on downstream tasks…
ANCRe: Adaptive Neural Connection Reassignment for Efficient Depth Scaling
Yilang Zhang, Bingcong Li, Niao He +1
Scaling network depth has been a central driver behind the success of modern foundation models, yet recent investigations suggest that deep layers are often underutilized. This pap…
ScaLoRA: Optimally Scaled Low-Rank Adaptation for Efficient High-Rank Fine-Tuning
Yilang Zhang, Xiaodong Yang, Yiwei Cai +1
As large language models (LLMs) continue to scale in size, the computational overhead has become a major bottleneck for task-specific fine-tuning. While low-rank adaptation (LoRA)…
VASSO: Variance Suppression for Sharpness-Aware Minimization
Bingcong Li, Yilang Zhang, Georgios B. Giannakis
Sharpness-aware minimization (SAM) has well-documented merits in enhancing generalization of deep neural network models. Accounting for sharpness in the loss function geometry, whe…
Learnable Loss Geometries with Mirror Descent for Scalable and Convergent Meta-Learning
Yilang Zhang, Bingcong Li, Georgios B. Giannakis
Utilizing task-invariant knowledge acquired from related tasks as prior information, meta-learning offers a principled approach to learning a new task with limited data records. Sa…