5 papers
Scalable Derivative Gaussian Processes via Exact Gradient Reduction
Hyunseok Seung, Matthias Katzfuss
Gradient observations can substantially improve Gaussian process (GP) surrogates, particularly in high-dimensional settings where function evaluations are expensive. However, exact…
Low-Rank Curvature for Zeroth-Order Optimization in LLM Fine-Tuning
Hyunseok Seung, Jaewoo Lee, Hyunsuk Ko
We introduce LOREN, a curvature-aware zeroth-order (ZO) optimization method for fine-tuning large language models (LLMs). Existing ZO methods, which estimate gradients via finite d…
MAC: An Efficient Gradient Preconditioning using Mean Activation Approximated Curvature
Hyunseok Seung, Jaewoo Lee, Hyunsuk Ko
Second-order optimization methods for training neural networks, such as KFAC, exhibit superior convergence by utilizing curvature information of loss landscape. However, it comes a…
NysAct: A Scalable Preconditioned Gradient Descent using Nystrom Approximation
Hyunseok Seung, Jaewoo Lee, Hyunsuk Ko
Adaptive gradient methods are computationally efficient and converge quickly, but they often suffer from poor generalization. In contrast, second-order methods enhance convergence…
An Adaptive Method Stabilizing Activations for Enhanced Generalization
Hyunseok Seung, Jaewoo Lee, Hyunsuk Ko
We introduce AdaAct, a novel optimization algorithm that adjusts learning rates according to activation variance. Our method enhances the stability of neuron outputs by incorporati…