4 papers
Flexora: Flexible Low Rank Adaptation for Large Language Models
Chenxing Wei, Yao Shu, Ying Tiffany He +1
Large Language Models (LLMs) are driving advancements in artificial intelligence by increasing the scale of model parameters, which has significantly enhanced generalization abilit…
Refining Adaptive Zeroth-Order Optimization at Ease
Yao Shu, Qixin Zhang, Kun He +1
Recently, zeroth-order (ZO) optimization plays an essential role in scenarios where gradient information is inaccessible or unaffordable, such as black-box systems and resource-con…
Ferret: Federated Full-Parameter Tuning at Scale for Large Language Models
Yao Shu, Wenyang Hu, See-Kiong Ng +2
Large Language Models (LLMs) have become indispensable in numerous real-world applications. However, fine-tuning these models at scale, especially in federated settings where data…
OptEx: Expediting First-Order Optimization with Approximately Parallelized Iterations
Yao Shu, Jiongfeng Fang, Ying Tiffany He +1
First-order optimization (FOO) algorithms are pivotal in numerous computational domains such as machine learning and signal denoising. However, their application to complex tasks l…