12 papers
Quantifying Hyperparameter Transfer and the Importance of Embedding Layer Learning Rate
Dayal Singh Kalra, Maissam Barkeshli
Hyperparameter transfer allows extrapolating optimal optimization hyperparameters from small to large scales, making it critical for training large language models (LLMs). This is…
Crystalline topological invariants in quantum many-body systems
Naren Manjunath, Maissam Barkeshli
Crystalline symmetries give rise to topological invariants that can distinguish quantum phases of matter. Understanding these in strongly interacting systems is an ongoing research…
Artificial Intelligence and the Structure of Mathematics
Maissam Barkeshli, Michael R. Douglas, Michael H. Freedman
Recent progress in artificial intelligence (AI) is unlocking transformative capabilities for mathematics. There is great hope that AI will help solve major open problems and autono…
Learning Pseudorandom Numbers with Transformers: Permuted Congruential Generators, Curricula, and Interpretability
Tao Tao, Maissam Barkeshli
We study the ability of Transformer models to learn sequences generated by Permuted Congruential Generators (PCGs), a widely used family of pseudo-random number generators (PRNGs).…
Soft symmetries of topological orders
Ryohei Kobayashi, Maissam Barkeshli
(2+1)D topological orders possess emergent symmetries given by a group , which consists of the braided tensor autoequivalences of the modular tensor catego…
On the origin of neural scaling laws: from random graphs to natural language
Maissam Barkeshli, Alberto Alfarano, Andrey Gromov
Scaling laws have played a major role in the modern AI revolution, providing practitioners predictive power over how the model performance will improve with increasing data, comput…