7 papers
Depth, Not Data: An Analysis of Hessian Spectral Bifurcation
Shenyang Deng, Boyao Liao, Zhuoli Ouyang +2
The eigenvalue distribution of the Hessian matrix plays a crucial role in understanding the optimization landscape of deep neural networks. Prior work has attributed the well-docum…
HTMuon: Improving Muon via Heavy-Tailed Spectral Correction
Tianyu Pang, Yujie Fang, Zihang Liu +4
Muon has recently shown promising results in LLM training. In this work, we study how to further improve Muon. We argue that Muon's orthogonalized update rule suppresses the emerge…
Suspicious Alignment of SGD: A Fine-Grained Step Size Condition Analysis
Shenyang Deng, Boyao Liao, Zhuoli Ouyang +3
This paper explores the suspicious alignment phenomenon in stochastic gradient descent (SGD) under ill-conditioned optimization, where the Hessian spectrum splits into dominant and…
LIFT the Veil for the Truth: Principal Weights Emerge after Rank Reduction for Reasoning-Focused Supervised Fine-Tuning
Zihang Liu, Tianyu Pang, Oleg Balabanov +5
Recent studies have shown that supervised fine-tuning of LLMs on a small number of high-quality datasets can yield strong reasoning capabilities. However, full fine-tuning (Full FT…
Eigenspectrum Analysis of Neural Networks without Aspect Ratio Bias
Yuanzhe Hu, Kinshuk Goel, Vlad Killiakov +1
Diagnosing deep neural networks (DNNs) by analyzing the eigenspectrum of their weights has been an active area of research in recent years. One of the main approaches involves meas…
KCES: Training-Free Defense for Robust Graph Neural Networks via Kernel Complexity
Yaning Jia, Shenyang Deng, Chiyu Ma +2
Graph Neural Networks (GNNs) have achieved impressive success across a wide range of graph-based tasks, yet they remain highly vulnerable to small, imperceptible perturbations and…