2 papers
cs.LG2026
Curvature-Weighted Gradient Diversity: A Noise Measure for Geometry-Adaptive SGD Schedules
Muhammad Hamza, Ayush Goel
The standard convergence analysis of mini-batch stochastic gradient descent (SGD) models gradient noise using a single variance term that treats all parameter directions equally, i…
cs.LG2026
In-Context Learning in Linear vs. Quadratic Attention Models: An Empirical Study on Regression Tasks
Ayush Goel, Arjun Kohli, Sarvagya Somvanshi
Recent work has demonstrated that transformers and linear attention models can perform in-context learning (ICL) on simple function classes, such as linear regression. In this pape…