2 citations · 2 across the 6 of their papers we have counts for
6 papers
Control Allocation in Neural Network Optimization: Joint Affine Control of Weight and Bias Updates
Zhang Gongyue, Sheng Yixuan, Wang Zhiyong +3
Optimization algorithms determine not only the magnitude of a neural-network update but also how that update is distributed across parameter channels. We study whether this distrib…
Hidden Boundary Motion in Transformer Optimization: Function-Space Orthogonalization of Affine Weight and Bias Updates
Zhang Gongyue, Sheng Yixuan, Liu donghan +3
Weights and biases are normally optimized as separate parameter tensors, yet they do not represent separate functions when the input to an affine layer has nonzero mean. For an aff…
The Anatomy of Implicit Bias: Information Allocation in Neural Network Training
Zhang Gongyue, Wang Zhiyong, Liu Donghan +3
Implicit bias is usually explained as the preference of an optimization process for certain final solutions and their geometry. This view helps explain where a model finally stops.…
Natural Spectral Fusion: p-Exponent Cyclic Scheduling and Early Decision-Boundary Alignment in First-Order Optimization
Gongyue Zhang, Honghai Liu
Spectral behaviors have been widely discussed in machine learning, yet the optimizer's own spectral bias remains unclear. We argue that first-order optimizers exhibit an intrinsic…
The Unified Balance Theory of Second-Moment Exponential Scaling Optimizers in Visual Tasks
Gongyue Zhang, Honghai Liu
We have identified a potential method for unifying first-order optimizers through the use of variable Second-Moment Exponential Scaling(SMES). We begin with back propagation, addre…
Asymmetric Momentum: A Rethinking of Gradient Descent
Gongyue Zhang, Dinghuang Zhang, Shuwen Zhao +3
Through theoretical and experimental validation, unlike all existing adaptive methods like Adam which penalize frequently-changing parameters and are only applicable to sparse grad…