Showing 2026Show all
3 papers · 1 filter
cs.LG2026
Learning Structured Sparsity for Large Models: Bridging Model Compressibility and Efficient Inference
Tianxiang Xia, Kai Lion, Niao He +3
Scaling laws continue to reward increases in model scale, while the resulting computational and memory demands drive greater reliance on cloud infrastructure for inference. Yet unr…
cs.LG2026
Muown Implicitly Performs Angular Step-size Decay
Florian Hübler, Kai Lion, Antonio Orvieto +1
Matrix-aware optimizers such as Muon and Muown have recently shown strong empirical performance for pre-training Transformers. In particular, Muown separates each weight matrix int…
cs.LG2026
Integrated electro-optic attention nonlinearities for transformers
Luis Mickeler, Kai Lion, Alfonso Nardi +5
Transformers achieve state-of-the-art performance in language processing and computer vision. Their computation increasingly depends on hardware that can keep pace not only with la…