4 citations · 5 across the 7 of their papers we have counts for
7 papers
(How) Learning Rates Regulate Catastrophic Overtraining
Mark Rofin, Aditya Varre, Nicolas Flammarion
Supervised fine-tuning (SFT) is a common first stage of LLM post-training, teaching the model to follow instructions and shaping its behavior as a helpful assistant. At the same ti…
Understanding the Emergence of Seemingly Useless Features in Next-Token Predictors
Mark Rofin, Jalal Naghiyev, Michael Hahn
Trained Transformers have been shown to compute abstract features that appear redundant for predicting the immediate next token. We identify which components of the gradient signal…
Gradient Flow Polarizes Softmax Outputs towards Low-Entropy Solutions
Aditya Varre, Mark Rofin, Nicolas Flammarion
Understanding the intricate non-convex training dynamics of softmax-based models is crucial for explaining the empirical success of transformers. In this article, we analyze the gr…
Lower Bounds for Chain-of-Thought Reasoning in Hard-Attention Transformers
Alireza Amiri, Xinting Huang, Mark Rofin +1
Chain-of-thought reasoning and scratchpads have emerged as critical tools for enhancing the computational capabilities of transformers. While theoretical results show that polynomi…
Why are Sensitive Functions Hard for Transformers?
Michael Hahn, Mark Rofin
Empirical studies have identified a range of learnability biases and limitations of transformers, such as a persistent difficulty in learning to compute simple formal languages suc…
Linear Interpolation In Parameter Space is Good Enough for Fine-Tuned Language Models
Mark Rofin, Nikita Balagansky, Daniil Gavrilov
The simplest way to obtain continuous interpolation between two points in high dimensional space is to draw a line between them. While previous works focused on the general connect…
Vote'n'Rank: Revision of Benchmarking with Social Choice Theory
Mark Rofin, Vladislav Mikhailov, Mikhail Florinskiy +5
The development of state-of-the-art systems in different applied areas of machine learning (ML) is driven by benchmarks, which have shaped the paradigm of evaluating generalisation…