6 papers
Statistical Inference for Rank Allocation in Low-Rank Adaptation
Yihang Gao, Vincent Y. F. Tan
Low-rank adaptation (LoRA) has become a widely used parameter-efficient fine-tuning method for large language models. Since different modules and layers may contribute unequally to…
FairGC: Fairness-aware Graph Condensation
Yihan Gao, Chenxi Huang, Wen Shi +5
Graph condensation (GC) has become a vital strategy for scaling Graph Neural Networks by compressing massive datasets into small, synthetic node sets. While current GC methods effe…
ODELoRA: Training Low-Rank Adaptation by Solving Ordinary Differential Equations
Yihang Gao, Vincent Y. F. Tan
Low-rank adaptation (LoRA) has emerged as a widely adopted parameter-efficient fine-tuning method in deep transfer learning, due to its reduced number of trainable parameters and l…
Automatic Rank Determination for Low-Rank Adaptation via Submodular Function Maximization
Yihang Gao, Vincent Y. F. Tan
In this paper, we propose SubLoRA, a rank determination method for Low-Rank Adaptation (LoRA) based on submodular function maximization. In contrast to prior approaches, such as Ad…
Low Tensor-Rank Adaptation of Kolmogorov--Arnold Networks
Yihang Gao, Michael K. Ng, Vincent Y. F. Tan
Kolmogorov--Arnold networks (KANs) have demonstrated their potential as an alternative to multi-layer perceptions (MLPs) in various domains, especially for science-related tasks. H…
On the Convergence of (Stochastic) Gradient Descent for Kolmogorov--Arnold Networks
Yihang Gao, Vincent Y. F. Tan
Kolmogorov--Arnold Networks (KANs), a recently proposed neural network architecture, have gained significant attention in the deep learning community, due to their potential as a v…