5 papers
Muown Implicitly Performs Angular Step-size Decay
Florian Hübler, Kai Lion, Antonio Orvieto +1
Matrix-aware optimizers such as Muon and Muown have recently shown strong empirical performance for pre-training Transformers. In particular, Muown separates each weight matrix int…
Muown: Row-Norm Control for Muon Optimization
Kai Lion, Florian Hübler, Bingcong Li +2
Muon has emerged as a strong competitor to AdamW for language model pre-training, yet its behavior at scale is sensitive to weight decay. Recent work has observed that, for Muon wi…
Integrated electro-optic attention nonlinearities for transformers
Luis Mickeler, Kai Lion, Alfonso Nardi +5
Transformers have emerged as the dominant neural-network architecture, achieving state-of-the-art performance in language processing and computer vision. At the core of these model…
PoLAR: Polar-Decomposed Low-Rank Adapter Representation
Kai Lion, Liang Zhang, Bingcong Li +1
We show that low-rank adaptation of large-scale models suffers from a low stable rank that is well below the linear algebraic rank of the subspace, degrading fine-tuning performanc…
Optical Computing with Spectrally Multiplexed Features in Complex Media
Xue Dong, Kai Lion, Fei Xia +4
Artificial intelligence (AI) has rapidly evolved into a critical technology; however, electrical hardware struggles to keep pace with the exponential growth of AI models. Free spac…