Mechanistic Insights into Grokking from the Embedding Layer

H. V. AlquBoj, Hilal AlQuabeh, Velibor Bojkovic +2

Grokking, a delayed generalization in neural networks after perfect training performance, has been observed in Transformers and MLPs, but the components driving it remain underexpl…