3 papers
cs.LG2026
Control Allocation in Neural Network Optimization: Joint Affine Control of Weight and Bias Updates
Zhang Gongyue, Sheng Yixuan, Wang Zhiyong +3
Optimization algorithms determine not only the magnitude of a neural-network update but also how that update is distributed across parameter channels. We study whether this distrib…
cs.LG2025
Natural Spectral Fusion: p-Exponent Cyclic Scheduling and Early Decision-Boundary Alignment in First-Order Optimization
Gongyue Zhang, Honghai Liu
Spectral behaviors have been widely discussed in machine learning, yet the optimizer's own spectral bias remains unclear. We argue that first-order optimizers exhibit an intrinsic…
cs.LG2024
The Unified Balance Theory of Second-Moment Exponential Scaling Optimizers in Visual Tasks
Gongyue Zhang, Honghai Liu
We have identified a potential method for unifying first-order optimizers through the use of variable Second-Moment Exponential Scaling(SMES). We begin with back propagation, addre…