3 citations · 3 across the 6 of their papers we have counts for
5 papers · 1 filter
How Does Orthogonalization Adapt to the Neural Network Hessian Structure? A Gradient Self Outer-Product Analysis at Initialization
Shenyang Deng, Garvit Goyal, Tianyu Pang +2
Muon orthogonalizes a weight matrix's momentum before each step, and on neural networks this simple preconditioner beats AdamW in a wide range of tasks. Existing analyses explain M…
Kernel-Complexity Edge Sanitization for Training-Free Defense against Structural Graph Attacks
Yaning Jia, Shenyang Deng, Yaoqing Yang +3
Graph Neural Networks (GNNs) have achieved remarkable success across diverse applications, yet they remain highly vulnerable to adversarial attacks that maliciously perturb graph s…
HTMuon: Improving Muon via Heavy-Tailed Spectral Correction
Tianyu Pang, Yujie Fang, Zihang Liu +4
Muon has recently shown promising results in LLM training. In this work, we study how to further improve Muon. We argue that Muon's orthogonalized update rule suppresses the emerge…
Depth, Not Data: An Analysis of Hessian Spectral Bifurcation
Shenyang Deng, Boyao Liao, Zhuoli Ouyang +2
The eigenvalue distribution of the Hessian matrix plays a crucial role in understanding the optimization landscape of deep neural networks. Prior work has attributed the well-docum…
Suspicious Alignment of SGD: A Fine-Grained Step Size Condition Analysis
Shenyang Deng, Boyao Liao, Zhuoli Ouyang +3
This paper explores the suspicious alignment phenomenon in stochastic gradient descent (SGD) under ill-conditioned optimization, where the Hessian spectrum splits into dominant and…