Tensor Programs IVb: Adaptive Optimization in the Infinite-Width Limit

Greg Yang, Etai Littwin

Going beyond stochastic gradient descent (SGD), what new phenomena emerge in wide neural networks trained by adaptive optimizers like Adam? Here we show: The same dichotomy between…