384 citations · 2.3k across the 48 of their papers we have counts for
Showing 2023 · cs.LGShow all
3 papers · 2 filters
cs.LG2023★ 8 cited
A Theory for Emergence of Complex Skills in Language Models
Sanjeev Arora, Anirudh Goyal
A major driver of AI products today is the fact that new skills emerge in language models when their parameter set and training corpora are scaled up. This phenomenon is poorly und…
cs.LG2023★ 36 cited
Fine-Tuning Language Models with Just Forward Passes
Sadhika Malladi, Tianyu Gao, Eshaan Nichani +4
Fine-tuning language models (LMs) has yielded success on diverse downstream tasks, but as LMs grow in size, backpropagation requires a prohibitively large amount of memory. Zeroth-…
cs.LG2023
Why (and When) does Local SGD Generalize Better than SGD?
Xinran Gu, Kaifeng Lyu, Longbo Huang +1
Local SGD is a communication-efficient variant of SGD for large-scale training, where multiple GPUs perform SGD independently and average the model parameters periodically. It has…