5 citations · 10 across the 15 of their papers we have counts for
5 papers · 1 filter
Hyperparameter Transfer in Graph Neural Networks
Gage DeZoort, Boris Hanin
The performance of deep learning models crucially depends on the settings of hyperparameters like learning rate, initialization scale, and weight decay. Hyperparameter transfer aim…
Bayesian Inference with Shaped Deep Non-linear MLPs
Boris Hanin, Tianze Jiang
A central aim of deep learning theory is to characterize how neural networks make predictions in the regime of simultaneously large model and training set size. Since the limits of…
Hyperparameter Transfer for Dense Associative Memories
Roi Holtzman, Dmitry Krotov, Boris Hanin
Dense Associative Memory (DenseAM) is a promising family of AI architectures that is represented by a neural network performing temporal dynamics on an energy landscape. While hype…
Learning Rate Transfer in Normalized Transformers
Boris Shigida, Boris Hanin, Andrey Gromov
The Normalized Transformer, or nGPT (arXiv:2410.01131) achieves impressive training speedups and does not require weight decay or learning rate warmup. However, despite having hype…
Hyperparameter Transfer with Mixture-of-Expert Layers
Tianze Jiang, Blake Bordelon, Cengiz Pehlevan +1
Mixture-of-Experts (MoE) layers have emerged as an important tool in scaling up modern neural networks by decoupling total trainable parameters from activated parameters in the for…