95 citations · 111 across the 4 of their papers we have counts for
4 papers
Normalization Layer Per-Example Gradients are Sufficient to Predict Gradient Noise Scale in Transformers
Gavia Gray, Aman Tiwari, Shane Bergsma +1
Per-example gradient norms are a vital ingredient for estimating gradient noise scale (GNS) with minimal variance. Observing the tensor contractions required to compute them, we pr…
Separable Layers Enable Structured Efficient Linear Substitutions
Gavin Gray, Elliot J. Crowley, Amos Storkey
In response to the development of recent efficient dense layers, this paper shows that something as simple as replacing linear components in pointwise convolutions with structured…
BlockSwap: Fisher-guided Block Substitution for Network Compression on a Budget
Jack Turner, Elliot J. Crowley, Michael O'Boyle +2
The desire to map neural networks to varying-capacity devices has led to the development of a wealth of compression techniques, many of which involve replacing standard convolution…
Moonshine: Distilling with Cheap Convolutions
Elliot J. Crowley, Gavin Gray, Amos Storkey
Many engineers wish to deploy modern neural networks in memory-limited settings; but the development of flexible methods for reducing memory use is in its infancy, and there is lit…