4k citations · 11.7k across the 27 of their papers we have counts for
Showing cs.LGShow all
2 papers · 1 filter
cs.LG2023★ 2 cited
A Theory on Adam Instability in Large-Scale Machine Learning
Igor Molybog, Peter Albert, Moya Chen +14
We present a theory for the previously unexplained divergent behavior noticed in the training of large language models. We argue that the phenomenon is an artifact of the dominant…
cs.LG2020★ 20 cited
Better Fine-Tuning by Reducing Representational Collapse
Armen Aghajanyan, Akshat Shrivastava, Anchit Gupta +3
Although widely adopted, existing approaches for fine-tuning pre-trained language models have been shown to be unstable across hyper-parameter settings, motivating recent work on t…