9 papers
The Geometric Structure of Models Learning Sparse Data
Thomas Walker, T. Mitchell Roddenberry, Ahmed Imtiaz Humayun +2
The manifold hypothesis (MH) is often used to explain how machine learning can overcome the curse of dimensionality. However, the MH is only applicable in regimes where the trainin…
The Linear Centroids Hypothesis: Features as Directions Learned by Local Experts
Thomas Walker, Ahmed Imtiaz Humayun, Randall Balestriero +1
The Linear Representation Hypothesis (LRH) identifies features of a trained deep network (DN) as linear directions in the activation spaces, i.e., output spaces of intermediate lay…
GrokAlign: Geometric Characterisation and Acceleration of Grokking
Thomas Walker, Ahmed Imtiaz Humayun, Randall Balestriero +1
A key challenge for the machine learning community is to understand and accelerate the training dynamics of deep networks that lead to delayed generalisation and emergent robustnes…
Max-Affine Spline Insights Into Deep Network Pruning
Haoran You, Randall Balestriero, Zhihan Lu +6
In this paper, we study the importance of pruning in Deep Networks (DNs) and the yin & yang relationship between (1) pruning highly overparametrized DNs that have been trained from…
Mitigating over-exploration in latent space optimization using LES
Omer Ronen, Ahmed Imtiaz Humayun, Richard Baraniuk +2
We develop Latent Exploration Score (LES) to mitigate over-exploration in Latent Space Optimization (LSO), a popular method for solving black-box discrete optimization problems. LS…
On the Geometry of Deep Learning
Randall Balestriero, Ahmed Imtiaz Humayun, Richard Baraniuk
In this paper, we overview one promising avenue of progress at the mathematical foundation of deep learning: the connection between deep networks and function approximation by affi…