4 papers
TLoRA: Task-aware Low Rank Adaptation of Large Language Models
Weicheng Lin, Yi Zhang, Jiawei Dang +1
Low-Rank Adaptation (LoRA) has become a widely adopted parameter-efficient fine-tuning method for large language models, with its effectiveness largely influenced by the allocation…
Mild Over-Parameterization Benefits Asymmetric Tensor PCA
Shihong Ding, Weicheng Lin, Cong Fang
Asymmetric Tensor PCA (ATPCA) is a prototypical model for studying the trade-offs between sample complexity, computation, and memory. Existing algorithms for this problem typically…
Learning Curves of Stochastic Gradient Descent in Kernel Regression
Haihan Zhang, Weicheng Lin, Yuanshi Liu +1
This paper considers a canonical problem in kernel regression: how good are the model performances when it is trained by the popular online first-order algorithms, compared to the…
An Effective Dynamic Gradient Calibration Method for Continual Learning
Weichen Lin, Jiaxiang Chen, Ruomin Huang +1
Continual learning (CL) is a fundamental topic in machine learning, where the goal is to train a model with continuously incoming data and tasks. Due to the memory limit, we cannot…