6 papers
Modeling with Categorical Features via Exact Fusion and Sparsity Regularisation
Kayhan Behdin, Riade Benbaki, Peter Radchenko +1
We study the high-dimensional linear regression problem with categorical predictors that have many levels. We propose a new estimation approach, which performs model compression vi…
Multi-Task Learning for Sparsity Pattern Heterogeneity: Statistical and Computational Perspectives
Kayhan Behdin, Gabriel Loewinger, Kenneth T. Kishida +2
We consider a problem in Multi-Task Learning (MTL) where multiple linear models are jointly trained on a collection of datasets ("tasks"). A key novelty of our framework is that it…
Differentially Private High-dimensional Variable Selection via Integer Programming
Petros Prastakos, Kayhan Behdin, Rahul Mazumder
Sparse variable selection improves interpretability and generalization in high-dimensional learning by selecting a small subset of informative features. Recent advances in Mixed In…
TSENOR: Highly-Efficient Algorithm for Finding Transposable N:M Sparse Masks
Xiang Meng, Mehdi Makni, Rahul Mazumder
Network pruning reduces the computational requirements of large neural networks, with N:M sparsity -- retaining only N out of every M consecutive weights -- offering a compelling b…
An Optimization Framework for Differentially Private Sparse Fine-Tuning
Mehdi Makni, Kayhan Behdin, Gabriel Afriat +5
Differentially private stochastic gradient descent (DP-SGD) is broadly considered to be the gold standard for training and fine-tuning neural networks under differential privacy (D…
HASSLE-free: A unified Framework for Sparse plus Low-Rank Matrix Decomposition for LLMs
Mehdi Makni, Kayhan Behdin, Zheng Xu +2
The impressive capabilities of large foundation models come at a cost of substantial computing resources to serve them. Compressing these pre-trained models is of practical interes…