5 papers
Measuring Dead Directions: Decomposing and Classifying Singular Structure off Canonical Alignment
Tejas Pradeep Shirodkar
We give a descent-free, alignment-free measurement of singular structure on trained networks. At a single frozen checkpoint the read recovers the order of each dead direction f…
Dead-Direction Conditioners: Gauge-Equivariant Preconditioning for Deep Networks
Tejas Pradeep Shirodkar
A deep network's loss is invariant to continuous symmetries of its parameters: the logit shift, the ReLU rescaling, the LayerNorm scale, the per-head attention rotation. Adam's per…
Dead-Direction Signatures: A Cheap Spectral Reading of Singular Complexity
Tejas Pradeep Shirodkar, P. J. Narayanan
Singular learning theory characterises the complexity of a deep network through the geometry of its loss singularities. The local learning coefficient (LLC), the standard estimator…
Algebraic Dead Directions in LayerNorm Transformers: A Forward-Pass-Only Diagnostic at LLM Scale
Tejas Pradeep Shirodkar, P. J. Narayanan
Pretrained transformers sit near singular minima of the loss, where the Fisher information metric degenerates along dead directions: directions in parameter space along which the d…
Dead Directions: Geometric Singular Learning
Tejas Pradeep Shirodkar
Singular learning theory and information geometry have studied the same parameter spaces in mostly separate vocabularies: the former computes Bayesian invariants in resolved coordi…