Universally Empowering Zeroth-Order Optimization via Adaptive Layer-wise Sampling

Fei Wang, Li Shen, Liang Ding +3

Zeroth-Order optimization presents a promising memory-efficient paradigm for fine-tuning Large Language Models by relying solely on forward passes. However, its practical adoption…