261 citations · 1.1k across the 24 of their papers we have counts for
24 papers
Tensor Programs IVb: Adaptive Optimization in the Infinite-Width Limit
Greg Yang, Etai Littwin
Going beyond stochastic gradient descent (SGD), what new phenomena emerge in wide neural networks trained by adaptive optimizers like Adam? Here we show: The same dichotomy between…
Width and Depth Limits Commute in Residual Networks
Soufiane Hayou, Greg Yang
We show that taking the width and depth to infinity in a deep neural network with skip connections, when branches are scaled by (the only nontrivial scaling), resu…
Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer
Greg Yang, Edward J. Hu, Igor Babuschkin +7
Hyperparameter (HP) tuning in deep learning is an expensive process, prohibitively so for neural networks (NNs) with billions of parameters. We show that, in the recently discovere…
Feature Learning in Infinite-Width Neural Networks
Greg Yang, Edward J. Hu
As its width tends to infinity, a deep neural network's behavior under gradient descent can become simplified and predictable (e.g. given by the Neural Tangent Kernel (NTK)), if it…
Tensor Programs III: Neural Matrix Laws
Greg Yang
In a neural network (NN), *weight matrices* linearly transform inputs into *preactivations* that are then transformed nonlinearly into *activations*. A typical NN interleaves multi…
Tensor Programs II: Neural Tangent Kernel for Any Architecture
Greg Yang
We prove that a randomly initialized neural network of *any architecture* has its Tangent Kernel (NTK) converge to a deterministic limit, as the network widths tend to infinity. We…