7 papers · 1 filter
OmniOpt: Taxonomy, Geometry, and Benchmarking of Modern Optimizers
Siyuan Li, Jiabao Pan, Yumou Liu +9
Optimizer selection for large-scale model training has become a system-level design decision constrained jointly by compute, memory, tuning budget, and task diversity, yet the land…
RMNP: Row-Momentum Normalized Preconditioning for Scalable Matrix-Based Optimization
Shenyang Deng, Zhuoli Ouyang, Tianyu Pang +4
Preconditioned adaptive methods have gained significant attention for training deep neural networks, as they capture rich curvature information of the loss landscape. The central c…
VecFormer: Towards Efficient and Generalizable Graph Transformer with Graph Token Attention
Jingbo Zhou, Jun Xia, Siyuan Li +11
Graph Transformer has demonstrated impressive capabilities in the field of graph representation learning. However, existing approaches face two critical challenges: (1) most models…
Depth, Not Data: An Analysis of Hessian Spectral Bifurcation
Shenyang Deng, Boyao Liao, Zhuoli Ouyang +2
The eigenvalue distribution of the Hessian matrix plays a crucial role in understanding the optimization landscape of deep neural networks. Prior work has attributed the well-docum…
Understanding Generalization from Embedding Dimension and Distributional Convergence
Junjie Yu, Zhuoli Ouyang, Haotian Deng +5
Deep neural networks often generalize well despite heavy over-parameterization, challenging classical parameter-based analyses. We study generalization from a representation-centri…
Suspicious Alignment of SGD: A Fine-Grained Step Size Condition Analysis
Shenyang Deng, Boyao Liao, Zhuoli Ouyang +3
This paper explores the suspicious alignment phenomenon in stochastic gradient descent (SGD) under ill-conditioned optimization, where the Hessian spectrum splits into dominant and…
Doloris: Dual Conditional Diffusion Implicit Bridges with Sparsity Masking Strategy for Unpaired Single-Cell Perturbation Estimation
Changxi Chi, Jun Xia, Yufei Huang +9
Estimating single-cell responses across various perturbations facilitates the identification of key genes and enhances drug screening, significantly boosting experimental efficienc…