5 citations · 5 across the 4 of their papers we have counts for
5 papers · 1 filter
Grokking Finite-Dimensional Algebra
Pascal Jr Tikeng Notsawo, Guillaume Dumas, Guillaume Rabusseau
This paper investigates the grokking phenomenon, which refers to the sudden transition from a long memorization to generalization observed during neural networks training, in the c…
Grokking Beyond the Euclidean Norm of Model Parameters
Pascal Jr Tikeng Notsawo, Guillaume Dumas, Guillaume Rabusseau
Grokking refers to a delayed generalization following overfitting when optimizing artificial neural networks with gradient-based methods. In this work, we demonstrate that grokking…
Stochastic Average Gradient : A Simple Empirical Investigation
Pascal Junior Tikeng Notsawo
Despite the recent growth of theoretical studies and empirical successes of neural networks, gradient backpropagation is still the most widely used algorithm for training such netw…
Predicting Grokking Long Before it Happens: A look into the loss landscape of models which grok
Pascal Jr. Tikeng Notsawo, Hattie Zhou, Mohammad Pezeshki +2
This paper focuses on predicting the occurrence of grokking in neural networks, a phenomenon in which perfect generalization emerges long after signs of overfitting or memorization…
Adaptive Discrete Communication Bottlenecks with Dynamic Vector Quantization
Dianbo Liu, Alex Lamb, Xu Ji +4
Vector Quantization (VQ) is a method for discretizing latent representations and has become a major part of the deep learning toolkit. It has been theoretically and empirically sho…