6 citations · 9 across the 11 of their papers we have counts for
7 papers · 1 filter
Why Does Adaptive Batching Help LLM Pretraining? A Perspective from Unbounded Variance
Arda Fazla, Antesh Upadhyay, Ege C. Kaya +2
Increasing the batch size during training is a common practice in large language model (LLM) pretraining, yet the theoretical justification behind its success is not well understoo…
Unified High-Probability Analysis of Stochastic Variance-Reduced Estimation
Zhankun Luo, Antesh Upadhyay, M. Berk Sahin +3
Stochastic estimators are fundamental to large-scale optimization, where population quantities must be inferred from noisy oracle observations. Although influential methods such as…
Beyond Bounded Variance: Variance-Reduced Normalized Methods for Nonconvex Optimization under Blum-Gladyshev Noise
Antesh Upadhyay, Arda Fazla, Abolfazl Hashemi
We study nonconvex stochastic optimization under the Blum-Gladyshev (-0) noise model, where the stochastic gradient variance grows quadratically with the distance from…
Lower Bounds and Proximally Anchored SGD for Non-Convex Minimization Under Unbounded Variance
Arda Fazla, Ege C. Kaya, Antesh Upadhyay +1
Analysis of Stochastic Gradient Descent (SGD) and its variants typically relies on the assumption of uniformly bounded variance, a condition that frequently fails in practical non-…
RAMPAGE: RAndomized Mid-Point for debiAsed Gradient Extrapolation
Zhankun Luo, M. Berk Sahin, Antesh Upadhyay +2
A celebrated method for Variational Inequalities (VIs) is Extragradient (EG), which can be viewed as a standard discrete-time integration scheme. With this view in mind, in this pa…
First-Order Softmax Weighted Switching Gradient Method for Distributed Stochastic Minimax Optimization with Stochastic Constraints
Zhankun Luo, Antesh Upadhyay, Sang Bin Moon +1
This paper addresses the distributed stochastic minimax optimization problem subject to stochastic constraints. We propose a novel first-order Softmax-Weighted Switching Gradient m…
Variance Reduction for Stochastic Switching in Functionally Constrained Federated Optimization
Antesh Upadhyay, Sang Bin Moon, Abolfazl Hashemi +1
Federated learning (FL) is increasingly used in safety-critical and fairness-sensitive applications, where learning objectives must be optimized subject to functional constraints.…