activity
20232026
most citedStraight to Zero: Why Linearly Decaying the Learning Rate to Zero Works Best for LLMs

1 citations · 2 across the 9 of their papers we have counts for

collaborators
Showing 2024Show all

2 papers · 1 filter

cs.LG2024

Normalization Layer Per-Example Gradients are Sufficient to Predict Gradient Noise Scale in Transformers

Gavia Gray, Aman Tiwari, Shane Bergsma +1

Per-example gradient norms are a vital ingredient for estimating gradient noise scale (GNS) with minimal variance. Observing the tensor contractions required to compute them, we pr…

cs.LG2024

Sparse maximal update parameterization: A holistic approach to sparse training dynamics

Nolan Dey, Shane Bergsma, Joel Hestness

Several challenges make it difficult for sparse neural networks to compete with dense models. First, setting a large fraction of weights to zero impairs forward and gradient signal…