6 papers
The Geometry of Projection Heads: Conditioning, Invariance, and Collapse
Faris Chaudhry
We develop a geometric theory of projection heads in self-supervised learning by modeling the head as a trainable Riemannian metric on the backbone representation manifold. We show…
Feature Starvation as Geometric Instability in Sparse Autoencoders
Faris Chaudhry, Keisuke Yano, Anthea Monod
Sparse autoencoders (SAEs) are used to disentangle the dense, polysemantic internal representations of large language models (LLMs) into interpretable, monosemantic concepts. Howev…
Trajectory-Restricted Optimization Conditions and Geometry-Aware Linear Convergence
Faris Chaudhry, Anthea Monod, Keisuke Yano
Linear convergence of first-order methods is typically characterized by global optimization conditions whose constants reflect worst-case geometry of the ambient space. In high-dim…
Scaling Laws and Pathologies of Single-Layer PINNs: Network Width and PDE Nonlinearity
Faris Chaudhry
We establish empirical scaling laws for Single-Layer Physics-Informed Neural Networks on canonical nonlinear PDEs. We identify a dual optimization failure: (i) a baseline pathology…
Asymptotic and Finite-Time Guarantees for Langevin-Based Temperature Annealing in InfoNCE
Faris Chaudhry
The InfoNCE loss in contrastive learning depends critically on a temperature parameter, yet its dynamics under fixed versus annealed schedules remain poorly understood. We provide…
Implicit Statistical Inference in Transformers: Approximating Likelihood-Ratio Tests In-Context
Faris Chaudhry, Siddhant Gadkari
In-context learning (ICL) allows Transformers to adapt to novel tasks without weight updates, yet the underlying algorithms remain poorly understood. We adopt a statistical decisio…