Which Geometry on Which Layer? Measured Sign-Spectral Assignment in Mixed-Optimizer Training

Hiroki Naganuma, Laura Gomezjurado, Mahdi Ghaznavi +5

Modern neural-network training uses mixed recipes, assigning AdamW-like or sign-based updates to embedding and output layers and spectral updates to hidden matrices. These assignme…