Regularity-Aware Stochastic MGDA with Adaptive Per-Iterate Conflict-Avoidant Update

Abstract

Multi-objective learning (MOL) aims to optimize multiple objectives simultaneously. The multi-gradient descent algorithm (MGDA) constructs a common descent or conflict-avoidant (CA) direction by solving a gradient-aggregation subproblem. In stochastic settings, the vanilla stochastic MGDA method, SMG, replaces full-batch gradients with mini-batch estimates, causing bias in the resulting CA direction. Existing analyses control this error using continuity of the CA-direction mapping, but do not distinguish how its sensitivity varies with the geometry of the MGDA subproblem. Motivated by this limitation, we show that the MGDA direction map is Lipschitz continuous under reduced nondegeneracy, whereas its general \(1/2\)-Hölder exponent on bounded sets is sharp. This geometry motivates the Multi-objective Regularity-aware (MoRe) method, which retains adaptive MGDA aggregation in the regular regime and switches to a fixed-weight linear scalarization (LS) near degeneracy. The fallback branch skips the MGDA subproblem solve, which can reduce aggregation cost on fallback iterations while avoiding a potentially noise-sensitive update. Theoretically, we establish a per-iterate CA guarantee, and a \(\widetilde{\mathcal O}(T^{-1})\) convergence rate, improving upon the previously established \(\widetilde{\mathcal O}(T^{-1/4})\) SMG guarantee, where \(\widetilde{\mathcal O}\) hides logarithmic factors. Experiments demonstrate favorable multi-task performance and convergence behavior consistent with the theoretical rate.