2 citations · 5 across the 16 of their papers we have counts for
16 papers
Stabilizing Extrapolation in Looped Transformers via Learned Stochastic Stopping
Hsun-Yu Kuo, El Mahdi Chayti, Patrik Reizinger +2
Looped Transformers, which repeatedly apply a shared transformer block, are an architecturally natural fit for variable-length algorithmic tasks. Although they can exhibit strong l…
Stochastic Zeroth-Order Optimization Under Heavy-Tailed Noise
Taha El Bakkali, El Mahdi Chayti, Qiuyi Zhang +2
We study stochastic zeroth-order (ZO) optimization of smooth nonconvex objectives under heavy-tailed sample-gradient noise. This regime is motivated by empirical evidence that grad…
Stochastic Compositional Optimization via Hybrid Momentum Frank--Wolfe
El Mahdi Chayti, Mohamad Fakhouri
Stochastic compositional optimization minimizes over a compact convex set when the inner map is available only through noisy samples. We propose Hybrid Momentum St…
Nonsmooth Nonconvex Optimization from Noisy Comparisons: Unbiased Gap Estimation and the Price of Preference Feedback
Taha El Bakkali, El Mahdi Chayti, Omar Saadi
We study the minimization of -Lipschitz, possibly nonsmooth and nonconvex from noisy pairwise comparisons: comparing and returns a bit equal to with pro…
RanSOM: Second-Order Momentum with Randomized Scaling for Constrained and Unconstrained Optimization
El Mahdi Chayti, Mohamad Fakhouri
Second-order momentum corrects the lag of momentum behind the current gradient with a Hessian-vector product (HVP), but existing corrections need either a Lipschitz Hessian or extr…
-LoRA: Effective Fine-Tuning via Base Model Rescaling
Aymane El Firdoussi, El Mahdi Chayti, Mohamed El Amine Seddik +1
Fine-tuning has proven to be highly effective in adapting pre-trained models to perform better on new desired tasks with minimal data samples. Among the most widely used approaches…
Stochastic Difference-of-Convex Optimization with Momentum
El Mahdi Chayti, Martin Jaggi
We study the online stochastic difference-of-convex (DC) program , which covers sparse learning with nonconvex penalties, rob…
Second-Order Optimization with Asynchronous Curvature
El Mahdi Chayti, Martin Jaggi
Second-order methods make fast progress per iteration, but forming and factorizing the Hessian often dominates their wall-clock time. We study \emph{Split-Client}, a task-parallel…
Stochastic Optimization with Random Search
El Mahdi Chayti, Taha El Bakkali El Kadi, Omar Saadi +1
We revisit random search for stochastic optimization, where only noisy function evaluations are available. We show that the method works under weaker smoothness assumptions than pr…
Faster Gradient Methods for Highly-Smooth Stochastic Bilevel Optimization
Lesi Chen, Junru Li, El Mahdi Chayti +1
This paper studies the complexity of finding an -stationary point for stochastic bilevel optimization when the upper-level problem is nonconvex and the lower-level problem is st…
Improving Stochastic Cubic Newton with Momentum
El Mahdi Chayti, Nikita Doikov, Martin Jaggi
We study stochastic second-order methods for solving general non-convex optimization problems. We propose using a special version of momentum to stabilize the stochastic gradient a…
A New First-Order Meta-Learning Algorithm with Convergence Guarantees
El Mahdi Chayti, Martin Jaggi
Learning new tasks by leveraging prior experience is a fundamental trait of intelligent systems. While Model-Agnostic Meta-Learning (MAML) is a leading approach, it suffers from si…
Unified Convergence Theory of Stochastic and Variance-Reduced Cubic Newton Methods
El Mahdi Chayti, Nikita Doikov, Martin Jaggi
We study stochastic Cubic Newton methods for solving general possibly non-convex minimization problems. We propose a new framework, which we call the helper framework, that provide…
Second-order optimization with lazy Hessians
Nikita Doikov, El Mahdi Chayti, Martin Jaggi
We analyze Newton's method with lazy Hessian updates for solving general possibly non-convex optimization problems. We propose to reuse a previously seen Hessian for several iterat…
Optimization with Access to Auxiliary Information
El Mahdi Chayti, Sai Praneeth Karimireddy
We investigate the fundamental optimization question of minimizing a target function , whose gradients are expensive to compute or have limited availability, given access to som…
Linear Speedup in Personalized Collaborative Learning
El Mahdi Chayti, Sai Praneeth Karimireddy, Sebastian U. Stich +2
Collaborative training can improve the accuracy of a model for a user by trading off the model's bias (introduced by using data from other users who are potentially different) agai…