1 citations · 1 across the 6 of their papers we have counts for
6 papers
Learning Rate Transfer in Normalized Transformers
Boris Shigida, Boris Hanin, Andrey Gromov
The Normalized Transformer, or nGPT (arXiv:2410.01131) achieves impressive training speedups and does not require weight decay or learning rate warmup. However, despite having hype…
MobileLLM-Flash: Latency-Guided On-Device LLM Design for Industry Scale Deployment
Hanxian Huang, Igor Fedorov, Andrey Gromov +14
Real-time AI experiences call for on-device large language models (OD-LLMs) optimized for efficient deployment on resource-constrained hardware. The most useful OD-LLMs produce nea…
Towards Distributed Neural Architectures
Aditya Cowsik, Tianyu He, Andrey Gromov
We introduce and train distributed neural architectures (DNA) in vision and language domains. DNAs are initialized with a proto-architecture that consists of (transformer, MLP, att…
Grokking Modular Polynomials
Darshil Doshi, Tianyu He, Aritra Das +1
Neural networks readily learn a subset of the modular arithmetic tasks, while failing to generalize on the rest. This limitation remains unmoved by the choice of architecture and t…
Learning to grok: Emergence of in-context learning and skill composition in modular arithmetic tasks
Tianyu He, Darshil Doshi, Aritra Das +1
Large language models can solve tasks that were not present in the training set. This capability is believed to be due to in-context learning and skill composition. In this work, w…
To grok or not to grok: Disentangling generalization and memorization on corrupted algorithmic datasets
Darshil Doshi, Aritra Das, Tianyu He +1
Robust generalization is a major challenge in deep learning, particularly when the number of trainable parameters is very large. In general, it is very difficult to know if the net…