861 citations · 4.9k across the 19 of their papers we have counts for
19 papers
Feature Learning and Generalization in Deep Networks with Orthogonal Weights
Hannah Day, Yonatan Kahn, Daniel A. Roberts
Fully-connected deep neural networks with weights initialized from independent Gaussian distributions can be tuned to criticality, which prevents the exponential growth or decay of…
A Solvable Model of Neural Scaling Laws
Alexander Maloney, Daniel A. Roberts, James Sully
Large language models with a huge number of parameters, when trained on near internet-sized number of tokens, have been empirically shown to obey neural scaling laws: specifically,…
The Principles of Deep Learning Theory
Daniel A. Roberts, Sho Yaida, Boris Hanin
This book develops an effective theory approach to understanding deep neural networks of practical relevance. Beginning from a first-principles component-level picture of networks,…
SGD Implicitly Regularizes Generalization Error
Daniel A. Roberts
We derive a simple and model-independent formula for the change in the generalization gap due to a gradient descent update. We then compare the change in the test error for stochas…
Why is AI hard and Physics simple?
Daniel A. Roberts
We discuss why AI is hard and why physics is simple. We discuss how physical intuition and the approach of theoretical physics can be brought to bear on the field of artificial int…
Topological Obstructions to Autoencoding
Joshua Batson, C. Grace Haaf, Yonatan Kahn +1
Autoencoders have been proposed as a powerful tool for model-independent anomaly detection in high-energy physics. The operating principle is that events which do not belong to the…