Flat Loss, Evolving Features: Grokking in Modular Addition

Akash Kumar, Karthik Sankaranarayanan

Can a network keep learning while its training cross-entropy stays nearly flat at a positive level? We show that it can. On modular addition, a standard testbed for grokking, ReLU…