4 papers
Muon in Vision Transformers: Optimizer-Recipe Interactions and Gradient Spectra
Ben S. Southworth, Shuai Jiang, Daniel McBride +2
Muon is a recently developed matrix-aware optimizer that has shown strong results in transformer training, but its behavior in vision transformers (ViTs) is not yet well understood…
Cascade Token Selection for Transformer Attention Acceleration
Stephen J. Thomas
A method is presented for reducing the cost of representative token selection in transformer attention layers by exploiting the coherence of the representative set across depth. Ac…
Gated Subspace Inference for Transformer Acceleration
Stephen J. Thomas
A method is presented for accelerating inference in transformer language models by exploiting the low effective rank of the token activation manifold at each layer. The method deco…
Beyond Muon: MUD (MomentUm Decorrelation) for Faster Transformer Training
Ben S. Southworth, Stephen Thomas
Orthogonalized-momentum optimizers such as Muon improve transformer training by approximately whitening/orthogonalizing matrix-valued momentum updates via a short polar-decompositi…