3 papers
cs.LG2026
AF-Muon: An AdamW-Free Muon Optimizer for Tied-Embedding Models
Arash Lagzian, Paniz Halvachi, Junming Zhang +2
Muon improves large-scale training by applying a spectral-norm steepest-descent update to matrix parameters, but practical models also contain parameter blocks that do not fit dens…
math.OC2024
Randomized Block Coordinate DC Programming
Hoomaan Maskan, Paniz Halvachi, Suvrit Sra +1
We introduce an extension of the Difference of Convex Algorithm (DCA) in the form of a randomized block coordinate approach for problems with separable structure. For coordinat…
cs.LG2023
ELSA: Partial Weight Freezing for Overhead-Free Sparse Network Deployment
Paniz Halvachi, Alexandra Peste, Dan Alistarh +1
We present ELSA, a practical solution for creating deep networks that can easily be deployed at different levels of sparsity. The core idea is to embed one or more sparse networks…