Showing cs.LGShow all
2 papers · 1 filter
cs.LG2025
Deconstructing Positional Information: From Attention Logits to Training Biases
Zihan Gu, Ruoyu Chen, Han Zhang +2
Positional encodings enable Transformers to incorporate sequential information, yet their theoretical understanding remains limited to two properties: distance attenuation and tran…
cs.LG2025
Beyond Progress Measures: Theoretical Insights into the Mechanism of Grokking
Zihan Gu, Ruoyu Chen, Hua Zhang +2
Grokking, referring to the abrupt improvement in test accuracy after extended overfitting, offers valuable insights into the mechanisms of model generalization. Existing researches…