4 citations · 17 across the 25 of their papers we have counts for
30 papers
Reconstruction-Optimized Expert Pruning for Mixture-of-Experts Language Models
Jelena Markovic-Voronov, Petros Prastakos, Kayhan Behdin +7
Mixture-of-Experts (MoE) models reduce computation through sparse expert activation, but deployment still requires storing a large number of expert parameters. We study post-traini…
Maximum-modulus points on concentric circles
Yunpeng Li
We give a negative answer to the second question of Erdős Problem #1117: can the number of maximum-modulus points of a nonzero, non-monomial entire function on ten…
Smaller Models, Better Rejects: Preference Distillation Scaling
Rui Cai, Wenhui Zhu, Xiwen Chen +15
Preference distillation typically treats a teacher response as preferred and the student's own response as rejected. This assumes that self-generated failures are the most informat…
TREK: Distill to Explore, Reinforce to Refine
Yuanda Xu, Zhengze Zhou, Kayhan Behdin +10
Group Relative Policy Optimization (GRPO) is effective when the current policy already samples useful reasoning trajectories, but it stalls on hard prompts whose correct solution m…
Sampling for Quality: Training-Free Reward-Guided LLM Decoding via Sequential Monte Carlo
Jelena Markovic-Voronov, Wenhui Zhu, Bo Long +5
We introduce a principled probabilistic framework for reward-guided decoding in large language models, addressing the limitations of standard decoding methods that optimize token-l…
Closed-Form Formulas for Designing Ultra-Low Phase-Noise Cross-Coupled Dynamically Body-Biased Only-NMOS LCVCOs
Naser Khatti Dizabadi, Peter LoPresti
This paper presents a system-level analytical framework for modeling and minimizing phase noise in body-biased cross-coupled LC-tank voltage-controlled oscillators (LC-VCOs). Build…
Robust Batch-Level Query Routing for Large Language Models under Cost and Capacity Constraints
Jelena Markovic-Voronov, Kayhan Behdin, Yuanda Xu +3
We study the problem of routing queries to large language models (LLMs) under cost, GPU resources, and concurrency constraints. Prior per-query routing methods often fail to contro…
Semantic Search At LinkedIn
Fedor Borisyuk, Sriram Vasudevan, Muchen Wu +71
Semantic search with large language models (LLMs) enables retrieval by meaning rather than keyword overlap, but scaling it requires major inference efficiency advances. We present…
MixLM: High-Throughput and Effective LLM Ranking via Text-Embedding Mix-Interaction
Guoyao Li, Ran He, Shusen Jing +21
Large language models (LLMs) excel at capturing semantic nuances and therefore show impressive relevance ranking performance in modern recommendation and search systems. However, t…
Scaling Up Efficient Small Language Models Serving and Deployment for Semantic Job Search
Kayhan Behdin, Qingquan Song, Sriram Vasudevan +17
Large Language Models (LLMs) have demonstrated impressive quality when applied to predictive tasks such as relevance ranking and semantic search. However, deployment of such LLMs r…
Differentially Private High-dimensional Variable Selection via Integer Programming
Petros Prastakos, Kayhan Behdin, Rahul Mazumder
Sparse variable selection improves interpretability and generalization in high-dimensional learning by selecting a small subset of informative features. Recent advances in Mixed In…
Reasoning Models Can be Accurately Pruned Via Chain-of-Thought Reconstruction
Ryan Lucas, Kayhan Behdin, Zhipeng Wang +3
Reasoning language models such as DeepSeek-R1 produce long chain-of-thought traces during inference time which make them costly to deploy at scale. We show that using compression t…
BP-Seg: A graphical model approach to unsupervised and non-contiguous text segmentation using belief propagation
Fengyi Li, Kayhan Behdin, Natesh Pillai +3
Text segmentation based on the semantic meaning of sentences is a fundamental task with broad utility in many downstream applications. In this paper, we propose a graphical model-b…
An Optimization Framework for Differentially Private Sparse Fine-Tuning
Mehdi Makni, Kayhan Behdin, Gabriel Afriat +5
Differentially private stochastic gradient descent (DP-SGD) is broadly considered to be the gold standard for training and fine-tuning neural networks under differential privacy (D…
HASSLE-free: A unified Framework for Sparse plus Low-Rank Matrix Decomposition for LLMs
Mehdi Makni, Kayhan Behdin, Zheng Xu +2
The impressive capabilities of large foundation models come at a cost of substantial computing resources to serve them. Compressing these pre-trained models is of practical interes…
Scaling Down, Serving Fast: Compressing and Deploying Efficient LLMs for Recommendation Systems
Kayhan Behdin, Ata Fatahibaarzi, Qingquan Song +17
Large language models (LLMs) have demonstrated remarkable performance across a wide range of industrial applications, from search and recommendation systems to generative tasks. Al…
ALPS: Improved Optimization for Highly Sparse One-Shot Pruning for Large Language Models
Xiang Meng, Kayhan Behdin, Haoyue Wang +1
The impressive performance of Large Language Models (LLMs) across various natural language processing tasks comes at the cost of vast computational resources and storage requiremen…
OSSCAR: One-Shot Structured Pruning in Vision and Language Models with Combinatorial Optimization
Xiang Meng, Shibal Ibrahim, Kayhan Behdin +3
Structured pruning is a promising approach for reducing the inference costs of large vision and language models. By removing carefully chosen structures, e.g., neurons or attention…
End-to-end Feature Selection Approach for Learning Skinny Trees
Shibal Ibrahim, Kayhan Behdin, Rahul Mazumder
We propose a new optimization-based approach for feature selection in tree ensembles, an important problem in statistics and machine learning. Popular tree ensemble toolkits e.g.,…
QuantEase: Optimization-based Quantization for Language Models
Kayhan Behdin, Ayan Acharya, Aman Gupta +4
With the rising popularity of Large Language Models (LLMs), there has been an increasing interest in compression techniques that enable their efficient deployment. This study focus…
Sparse Gaussian Graphical Models with Discrete Optimization: Computational and Statistical Perspectives
Kayhan Behdin, Wenyu Chen, Rahul Mazumder
We consider the problem of learning a sparse graph underlying an undirected Gaussian graphical model, a key problem in statistical machine learning. Given samples from a multiv…
On Statistical Properties of Sharpness-Aware Minimization: Provable Guarantees
Kayhan Behdin, Rahul Mazumder
Sharpness-Aware Minimization (SAM) is a recent optimization framework aiming to improve the deep neural network generalization, through obtaining flatter (i.e. less sharp) solution…
mSAM: Micro-Batch-Averaged Sharpness-Aware Minimization
Kayhan Behdin, Qingquan Song, Aman Gupta +7
Modern deep learning models are over-parameterized, where different optima can result in widely varying generalization performance. The Sharpness-Aware Minimization (SAM) technique…
Improved Deep Neural Network Generalization Using m-Sharpness-Aware Minimization
Kayhan Behdin, Qingquan Song, Aman Gupta +4
Modern deep learning models are over-parameterized, where the optimization setup strongly affects the generalization performance. A key element of reliable optimization for these s…