5 papers
Stabilizing Extrapolation in Looped Transformers via Learned Stochastic Stopping
Hsun-Yu Kuo, El Mahdi Chayti, Patrik Reizinger +2
Looped Transformers, which repeatedly apply a shared transformer block, are an architecturally natural fit for variable-length algorithmic tasks. Although they can exhibit strong l…
A Split-Client Approach to Second-Order Optimization
El Mahdi Chayti, Martin Jaggi
Second-order optimization methods offer superior convergence rates but are often bottlenecked by the wall-clock cost of Hessian computation and factorization. In the moderate-dimen…
Stochastic Optimization with Random Search
El Mahdi Chayti, Taha El Bakkali El Kadi, Omar Saadi +1
We revisit random search for stochastic optimization, where only noisy function evaluations are available. We show that the method works under weaker smoothness assumptions than pr…
-LoRA: Effective Fine-Tuning via Base Model Rescaling
Aymane El Firdoussi, El Mahdi Chayti, Mohamed El Amine Seddik +1
Fine-tuning has proven to be highly effective in adapting pre-trained models to perform better on new desired tasks with minimal data samples. Among the most widely used approaches…
Stochastic Difference-of-Convex Optimization with Momentum
El Mahdi Chayti, Martin Jaggi
Stochastic difference-of-convex (DC) optimization is prevalent in numerous machine learning applications, yet its convergence properties under small batch sizes remain poorly under…