540 citations · 4.8k across the 94 of their papers we have counts for
3 papers · 2 filters
Learning Pseudorandom Numbers with Transformers: Permuted Congruential Generators, Curricula, and Interpretability
Tao Tao, Maissam Barkeshli
We study the ability of Transformer models to learn sequences generated by Permuted Congruential Generators (PCGs), a widely used family of pseudo-random number generators (PRNGs).…
When Can You Get Away with Low Memory Adam?
Dayal Singh Kalra, John Kirchenbauer, Maissam Barkeshli +1
Adam is the go-to optimizer for training modern machine learning models, but it requires additional memory to maintain the moving averages of the gradients and their squares. While…
(How) Can Transformers Predict Pseudo-Random Numbers?
Tao Tao, Darshil Doshi, Dayal Singh Kalra +2
Transformers excel at discovering patterns in sequential data, yet their fundamental limitations and learning mechanisms remain crucial topics of investigation. In this paper, we s…