Variance-reduced Zeroth-Order Methods for Fine-Tuning Language Models

Tanmay Gautam, Youngsuk Park, Hao Zhou +2

Fine-tuning language models (LMs) has demonstrated success in a wide array of downstream tasks. However, as LMs are scaled up, the memory requirements for backpropagation become pr…