551 citations · 1.9k across the 23 of their papers we have counts for
23 papers
Small-scale proxies for large-scale Transformer training instabilities
Mitchell Wortsman, Peter J. Liu, Lechao Xiao +13
Teams that have trained large Transformer-based models have reported training instabilities at large scale that did not appear when training with the same hyperparameters at smalle…
Kernel Regression with Infinite-Width Neural Networks on Millions of Examples
Ben Adlam, Jaehoon Lee, Shreyas Padhy +2
Neural kernels have drastically increased performance on diverse and nonstandard data modalities but require significantly more compute, which previously limited their application…
Fast Neural Kernel Embeddings for General Activations
Insu Han, Amir Zandieh, Jaehoon Lee +3
Infinite width limit has shed light on generalization and optimization aspects of deep learning by establishing connections between neural networks and kernel methods. Despite thei…
Dataset Distillation with Infinitely Wide Convolutional Networks
Timothy Nguyen, Roman Novak, Lechao Xiao +1
The effectiveness of machine learning algorithms arises from being able to extract useful features from large amounts of data. As model and dataset sizes increase, dataset distilla…
Explaining Neural Scaling Laws
Yasaman Bahri, Ethan Dyer, Jared Kaplan +2
The population loss of trained deep neural networks often follows precise power-law scaling relations with either the size of the training dataset or the number of parameters in th…
Towards NNGP-guided Neural Architecture Search
Daniel S. Park, Jaehoon Lee, Daiyi Peng +2
The predictions of wide Bayesian neural networks are described by a Gaussian process, known as the Neural Network Gaussian Process (NNGP). Analytic forms for NNGP kernels are known…