Zero-Order Optimization for LLM Fine-Tuning via Learnable Direction Sampling

Valery Parfenov, Grigoriy Evseev, Andrey Veprikov +3

Fine-tuning large pretrained language models (LLMs) is a cornerstone of modern NLP, yet its growing memory demands (driven by backpropagation and large optimizer States) limit depl…