5 papers
ZO-Act: Efficient Zeroth-Order Fine-Tuning via One-Shot Activation-Informed Low-Rank Subspaces
Xun Dong, Yibo Xu, Naigang Wang +3
Zeroth-order (ZO) optimization enables fine-tuning large language models when backpropagation is unavailable or memory-prohibitive, but existing methods often perturb full model we…
Persistent Cross-Attempt State Optimization for Repository-Level Code Generation
Ruwei Pan, Jiangshuai Wang, Qisheng Zhang +6
Large language models (LLMs) have achieved substantial progress in repository-level code generation. However, solving the same repository-level task often requires multiple attempt…
Toward Executable Repository-Level Code Generation via Environment Alignment
Ruwei Pan, Junlei Shen, Linhao Wu +5
Large language models (LLMs) have achieved strong performance on code generation, but existing methods still struggle with repository-level code generation under executable validat…
DiaBlo: Diagonal Blocks Are Sufficient For Finetuning
Selcuk Gurses, Aozhong Zhang, Yanxia Deng +5
Fine-tuning is a critical step for adapting large language models (LLMs) to domain-specific downstream tasks. To mitigate the substantial computational and memory costs of full-mod…
CLoQ: Enhancing Fine-Tuning of Quantized LLMs via Calibrated LoRA Initialization
Yanxia Deng, Aozhong Zhang, Selcuk Gurses +3
Fine-tuning large language models (LLMs) using low-rank adaptation (LoRA) has become a highly efficient approach for downstream tasks, particularly in scenarios with limited comput…