4 papers
SPICE: Submodular Penalized Information-Conflict Selection for Efficient Large Language Model Training
Powei Chang, Jinpeng Zhang, Bowen Chen +9
Information-based data selection for instruction tuning is compelling: maximizing the log-determinant of the Fisher information yields a monotone submodular objective, enabling gre…
LoRA-MGPO: Mitigating Double Descent in Low-Rank Adaptation via Momentum-Guided Perturbation Optimization
Yupeng Chang, Chenlu Guo, Yi Chang +1
Parameter-efficient fine-tuning (PEFT), particularly Low-Rank Adaptation (LoRA), adapts large language models (LLMs) by training only a small fraction of parameters. However, as th…
CHBench: A Chinese Dataset for Evaluating Health in Large Language Models
Chenlu Guo, Nuo Xu, Yi Chang +1
With the rapid development of large language models (LLMs), assessing their performance on health-related inquiries has become increasingly essential. The use of these models in re…
NLoRA: Nyström-Initiated Low-Rank Adaptation for Large Language Models
Chenlu Guo, Yuan Wu, Yi Chang
Parameter-efficient fine-tuning (PEFT) is essential for adapting large language models (LLMs), with low-rank adaptation (LoRA) being the most popular approach. However, LoRA suffer…