6 papers
Geometry-Preserving Neural Architectures on Manifolds with Boundary
Karthik Elamvazhuthi, Shiba Biswal, Kian Rosenblum +4
A growing number of neural architectures have been proposed to enforce geometric constraints, including projection-based networks, exponential-map updates, constrained output layer…
Matricial Free Energy as a Gaussianizing Regularizer: Enhancing Autoencoders for Gaussian Code Generation
Rishi Sonthalia, Raj Rao Nadakuditi
We introduce a novel regularization scheme for autoencoders based on matricial free energy. Our approach defines a differentiable loss function in terms of the singular values of t…
Risk Phase Transitions in Spiked Regression: Alignment Driven Benign and Catastrophic Overfitting
Jiping Li, Rishi Sonthalia
This paper analyzes the generalization error of minimum-norm interpolating solutions in linear regression using spiked covariance data models. The paper characterizes how varying s…
Low Rank Gradients and Where to Find Them
Rishi Sonthalia, Michael Murray, Guido Montúfar
This paper investigates low-rank structure in the gradients of the training loss for two-layer neural networks while relaxing the usual isotropy assumptions on the training data an…
Generalization for Least Squares Regression With Simple Spiked Covariances
Jiping Li, Rishi Sonthalia
Random matrix theory has proven to be a valuable tool in analyzing the generalization of linear models. However, the generalization properties of even two-layer neural networks tra…
Universal Approximation of Mean-Field Models via Transformers
Shiba Biswal, Karthik Elamvazhuthi, Rishi Sonthalia
This paper investigates the use of transformers to approximate the mean-field dynamics of interacting particle systems exhibiting collective behavior. Such systems are fundamental…