1 citations · 1 across the 7 of their papers we have counts for
7 papers
Curvature-Conditioned Multiscale Momentum with Sphere Constraints for LLM Pretraining
Shuchen Zhu, Yuxin Fang, Mingze Wang +1
Pretraining accounts for a large fraction of the total computational cost in LLM training. However, noise-dominant gradients and the highly ill-conditioned loss landscape bring sev…
ReasFlow: Assisting Reasoning-Centric Scientific Discovery in Applied Mathematics via a Knowledge-Based Multi-Agent System
Yutong He, Daibo Li, Guohong Li +15
Recent advances in Large Language Models have fueled autonomous AI agents capable of tackling complex scientific tasks, yet existing automated research systems remain predominantly…
Negligible in Size, Significant in Effect: On Scale Vectors in Large Language Models
Mingze Wang, Shuchen Zhu, Yuxin Fang +3
Normalization layers in modern large language models (LLMs) consist of a deterministic normalization operation and a learnable scale vector. While the normalization operation has b…
Subspace Optimization for Efficient Federated Learning under Heterogeneous Data
Shuchen Zhu, Zhengyang Huang, Yuqi Xu +1
Federated learning increasingly operates in a large-model regime where communication, memory, and computation are all scarce. Typically, non-IID client data induce drift that degra…
Accelerating LLM Pre-Training through Flat-Direction Dynamics Enhancement
Shuchen Zhu, Rizhen Hu, Mingze Wang +4
Pre-training Large Language Models requires immense computational resources, making optimizer efficiency essential. The optimization landscape is highly anisotropic, with loss redu…
SPARKLE: A Unified Single-Loop Primal-Dual Framework for Decentralized Bilevel Optimization
Shuchen Zhu, Boao Kong, Songtao Lu +2
This paper studies decentralized bilevel optimization, in which multiple agents collaborate to solve problems involving nested optimization structures with neighborhood communicati…
Decentralized Bilevel Optimization: A Perspective from Transient Iteration Complexity
Boao Kong, Shuchen Zhu, Songtao Lu +2
Stochastic bilevel optimization (SBO) is becoming increasingly essential in machine learning due to its versatility in handling nested structures. To address large-scale SBO, decen…