4 citations · 17 across the 25 of their papers we have counts for
11 papers · 1 filter
Reconstruction-Optimized Expert Pruning for Mixture-of-Experts Language Models
Jelena Markovic-Voronov, Petros Prastakos, Kayhan Behdin +7
Mixture-of-Experts (MoE) models reduce computation through sparse expert activation, but deployment still requires storing a large number of expert parameters. We study post-traini…
Smaller Models, Better Rejects: Preference Distillation Scaling
Rui Cai, Wenhui Zhu, Xiwen Chen +15
Preference distillation typically treats a teacher response as preferred and the student's own response as rejected. This assumes that self-generated failures are the most informat…
TREK: Distill to Explore, Reinforce to Refine
Yuanda Xu, Zhengze Zhou, Kayhan Behdin +10
Group Relative Policy Optimization (GRPO) is effective when the current policy already samples useful reasoning trajectories, but it stalls on hard prompts whose correct solution m…
Sampling for Quality: Training-Free Reward-Guided LLM Decoding via Sequential Monte Carlo
Jelena Markovic-Voronov, Wenhui Zhu, Bo Long +5
We introduce a principled probabilistic framework for reward-guided decoding in large language models, addressing the limitations of standard decoding methods that optimize token-l…
Robust Batch-Level Query Routing for Large Language Models under Cost and Capacity Constraints
Jelena Markovic-Voronov, Kayhan Behdin, Yuanda Xu +3
We study the problem of routing queries to large language models (LLMs) under cost, GPU resources, and concurrency constraints. Prior per-query routing methods often fail to contro…
An Optimization Framework for Differentially Private Sparse Fine-Tuning
Mehdi Makni, Kayhan Behdin, Gabriel Afriat +5
Differentially private stochastic gradient descent (DP-SGD) is broadly considered to be the gold standard for training and fine-tuning neural networks under differential privacy (D…
ALPS: Improved Optimization for Highly Sparse One-Shot Pruning for Large Language Models
Xiang Meng, Kayhan Behdin, Haoyue Wang +1
The impressive performance of Large Language Models (LLMs) across various natural language processing tasks comes at the cost of vast computational resources and storage requiremen…
End-to-end Feature Selection Approach for Learning Skinny Trees
Shibal Ibrahim, Kayhan Behdin, Rahul Mazumder
We propose a new optimization-based approach for feature selection in tree ensembles, an important problem in statistics and machine learning. Popular tree ensemble toolkits e.g.,…
Sparse Gaussian Graphical Models with Discrete Optimization: Computational and Statistical Perspectives
Kayhan Behdin, Wenyu Chen, Rahul Mazumder
We consider the problem of learning a sparse graph underlying an undirected Gaussian graphical model, a key problem in statistical machine learning. Given samples from a multiv…
Improved Deep Neural Network Generalization Using m-Sharpness-Aware Minimization
Kayhan Behdin, Qingquan Song, Aman Gupta +4
Modern deep learning models are over-parameterized, where the optimization setup strongly affects the generalization performance. A key element of reliable optimization for these s…
Transduction with Matrix Completion Using Smoothed Rank Function
Ashkan Esmaeili, Kayhan Behdin, Mohammad Amin Fakharian +1
In this paper, we propose two new algorithms for transduction with Matrix Completion (MC) problem. The joint MC and prediction tasks are addressed simultaneously to enhance the acc…