Low Rank and Sparse Fourier Structure in Recurrent Networks Trained on Modular Addition

Akshay Rangamani

Modular addition tasks serve as a useful test bed for observing empirical phenomena in deep learning, including the phenomenon of \emph{grokking}. Prior work has shown that one-lay…