1 citations · 2 across the 4 of their papers we have counts for
5 papers · 1 filter
Low-Rank Curvature for Zeroth-Order Optimization in LLM Fine-Tuning
Hyunseok Seung, Jaewoo Lee, Hyunsuk Ko
We introduce LOREN, a curvature-aware zeroth-order (ZO) optimization method for fine-tuning large language models (LLMs). Existing ZO methods, which estimate gradients via finite d…
NysAct: A Scalable Preconditioned Gradient Descent using Nystrom Approximation
Hyunseok Seung, Jaewoo Lee, Hyunsuk Ko
Adaptive gradient methods are computationally efficient and converge quickly, but they often suffer from poor generalization. In contrast, second-order methods enhance convergence…
An Adaptive Method Stabilizing Activations for Enhanced Generalization
Hyunseok Seung, Jaewoo Lee, Hyunsuk Ko
We introduce AdaAct, a novel optimization algorithm that adjusts learning rates according to activation variance. Our method enhances the stability of neuron outputs by incorporati…
MAC: An Efficient Gradient Preconditioning using Mean Activation Approximated Curvature
Hyunseok Seung, Jaewoo Lee, Hyunsuk Ko
Second-order optimization methods for training neural networks, such as KFAC, exhibit superior convergence by utilizing curvature information of loss landscape. However, it comes a…
Harmony in Divergence: Towards Fast, Accurate, and Memory-efficient Zeroth-order LLM Fine-tuning
Qitao Tan, Jun Liu, Zheng Zhan +6
Large language models (LLMs) excel across various tasks, but standard first-order (FO) fine-tuning demands considerable memory, significantly limiting real-world deployment. Recent…