8 papers
Lower Bounds for Anytime Acceleration of Gradient Descent
Chung-En Tsai, Ilyas Fatkhullin, Liang Zhang +1
Recent work suggests that the convergence rate of gradient descent (GD) in smooth convex optimization can be significantly improved by employing large stepsizes that may violate th…
Global Optimality for Constrained Exploration via Penalty Regularization
Florian Wolf, Ilyas Fatkhullin, Niao He
Efficient exploration is a central problem in reinforcement learning and is often formalized as maximizing the entropy of the state-action occupancy measure. While unconstrained ma…
Global Solutions to Non-Convex Functional Constrained Problems with Hidden Convexity
Ilyas Fatkhullin, Niao He, Guanghui Lan +1
Constrained non-convex optimization is fundamentally challenging, as global solutions are generally intractable and constraint qualifications may not hold. However, in many applica…
Natural Gradient VI: Guarantees for Non-Conjugate Models
Fangyuan Sun, Ilyas Fatkhullin, Niao He
Stochastic Natural Gradient Variational Inference (NGVI) is a widely used method for approximating posterior distribution in probabilistic models. Despite its empirical success and…
Second-order Optimization under Heavy-Tailed Noise: Hessian Clipping and Sample Complexity Limits
Abdurakhmon Sadiev, Peter Richtárik, Ilyas Fatkhullin
Heavy-tailed noise is pervasive in modern machine learning applications, arising from data heterogeneity, outliers, and non-stationary stochastic environments. While second-order m…
Can SGD Handle Heavy-Tailed Noise?
Ilyas Fatkhullin, Florian Hübler, Guanghui Lan
Stochastic Gradient Descent (SGD) is a cornerstone of large-scale optimization, yet its theoretical behavior under heavy-tailed noise -- common in modern machine learning and reinf…