10 citations · 12 across the 8 of their papers we have counts for
3 papers · 1 filter
Understanding and Minimising Outlier Features in Neural Network Training
Bobby He, Lorenzo Noci, Daniele Paliotta +2
Outlier Features (OFs) are neurons whose activation magnitudes significantly exceed the average over a neural network's (NN) width. They are well known to emerge during standard tr…
Hallmarks of Optimization Trajectories in Neural Networks: Directional Exploration and Redundancy
Sidak Pal Singh, Bobby He, Thomas Hofmann +1
We propose a fresh take on understanding the mechanisms of neural networks by analyzing the rich directional structure of optimization trajectories, represented by their pointwise…
Super Consistency of Neural Network Landscapes and Learning Rate Transfer
Lorenzo Noci, Alexandru Meterez, Thomas Hofmann +1
Recently, there has been growing evidence that if the width and depth of a neural network are scaled toward the so-called rich feature learning limit (\mup and its depth extension)…