most citedAligning the Pretraining and Finetuning Objectives of Language Models

1 citations · 1 across the 2 of their papers we have counts for

collaborators
Showing cs.LGShow all

1 paper · 1 filter

cs.LG2025

Cautious Weight Decay

Lizhang Chen, Jonathan Li, Kaizhao Liang +6

We introduce Cautious Weight Decay (CWD), a one-line, optimizer-agnostic modification that applies weight decay only to parameter coordinates whose signs align with the optimizer u…