8 papers
MpSub: A Momentum -Dimensional Subspace Trust-Region Method for Derivative-Free Fine-Tuning of Large Language Models
Yuyang Wang, Haoyu Yao, Pengcheng Xie
Full-parameter fine-tuning of large language models has substantial memory costs because backpropagation stores activations and gradients. Zeroth-order optimization avoids this by…
TOBYQA: A Time-Augmented Model-Based Method for Derivative-Free Optimization under Noise and Temporal Drift
Haoyu Yao, Pengcheng Xie
Derivative-free optimization (DFO) is challenging when the observation channel varies over time and evaluations are noisy. Conventional model-based methods assume stationary observ…
Why and When Neural Networks Improve Local Approximation in Optimization
Chengkuo Bian, Pengcheng Xie
Published experience with neural surrogates in derivative-free optimisation is contradictory: the same family of models that cuts the evaluation count of one solver leaves another…
Parallel Model-Based Derivative-Free Optimization via Rank-Two KKT Updates
Donghan Wu, Pengcheng Xie
Derivative-free optimization (DFO) addresses unconstrained problems $\min_{\x\in\RR^n} f(\x)$ where is accessed only through a zeroth-order oracle. Model-based trust-region met…
Low-Rank KKT Updates and a Parallel Flipping Mechanism for Model-Based Derivative-Free Optimization
Donghan Wu, Pengcheng Xie
Model-based derivative-free optimization relies on quadratic interpolation, but maintaining these models typically requires linear system solves. We show that fo…
BUP-TR: Bayesian Underdetermined Projection Trust-Region Methods for Derivative-Free Optimization
Wei Hu, Pengcheng Xie, Ya-Xiang Yuan +1
Underdetermined quadratic interpolation is a central model-construction tool in model-based derivative-free trust-region methods: it limits sampling costs but leaves an affine fami…