1 citations · 1 across the 9 of their papers we have counts for
11 papers
How Does Orthogonalization Adapt to the Neural Network Hessian Structure? A Gradient Self Outer-Product Analysis at Initialization
Shenyang Deng, Garvit Goyal, Tianyu Pang +2
Muon orthogonalizes a weight matrix's momentum before each step, and on neural networks this simple preconditioner beats AdamW in a wide range of tasks. Existing analyses explain M…
RMNP: Row-Momentum Normalized Preconditioning for Scalable Matrix-Based Optimization
Shenyang Deng, Zhuoli Ouyang, Tianyu Pang +4
Preconditioned adaptive methods have gained significant attention for training deep neural networks, as they capture rich curvature information of the loss landscape. The central c…
HTMuon: Improving Muon via Heavy-Tailed Spectral Correction
Tianyu Pang, Yujie Fang, Zihang Liu +4
Muon has recently shown promising results in LLM training. In this work, we study how to further improve Muon. We argue that Muon's orthogonalized update rule suppresses the emerge…
Decentralized Nonconvex Optimization under Heavy-Tailed Noise: Normalization and Optimal Convergence
Shuhua Yu, Dusan Jakovetic, Soummya Kar
Heavy-tailed noise in nonconvex stochastic optimization has garnered increasing research interest, as empirical studies, including those on training attention models, suggest it is…
Distributed Sign Momentum with Local Steps for Training Transformers
Shuhua Yu, Ding Zhou, Cong Xie +4
Pre-training Transformer models is resource-intensive, and recent studies have shown that sign momentum is an efficient technique for training large-scale deep learning models, par…
Large Deviation Upper Bounds and Improved MSE Rates of Nonlinear SGD: Heavy-tailed Noise and Power of Symmetry
Aleksandar Armacki, Shuhua Yu, Dragana Bajovic +2
We study large deviation upper bounds and mean-squared error (MSE) guarantees of a general framework of nonlinear stochastic gradient methods in the online setting, in the presence…
Nonlinear Stochastic Gradient Descent and Heavy-tailed Noise: A Unified Framework and High-probability Guarantees
Aleksandar Armacki, Shuhua Yu, Pranay Sharma +4
We study high-probability convergence in online learning, in the presence of heavy-tailed noise. To combat the heavy tails, a general framework of nonlinear SGD methods is consider…
Smoothed Gradient Clipping and Error Feedback for Decentralized Optimization under Symmetric Heavy-Tailed Noise
Shuhua Yu, Dusan Jakovetic, Soummya Kar
Motivated by understanding and analysis of large-scale machine learning under heavy-tailed gradient noise, we study decentralized optimization with gradient clipping, i.e., in whic…
Revisiting Image Classifier Training for Improved Certified Robust Defense against Adversarial Patches
Aniruddha Saha, Shuhua Yu, Arash Norouzzadeh +2
Certifiably robust defenses against adversarial patches for image classifiers ensure correct prediction against any changes to a constrained neighborhood of pixels. PatchCleanser a…
Secure Distributed Optimization Under Gradient Attacks
Shuhua Yu, Soummya Kar
In this paper, we study secure distributed optimization against arbitrary gradient attack in multi-agent networks. In distributed optimization, there is no central server to coordi…
Dynamic Median Consensus Over Random Networks
Shuhua Yu, Yuan Chen, Soummya Kar
This paper studies the problem of finding the median of N distinct numbers distributed across networked agents. Each agent updates its estimate for the median from noisy local obse…