8 papers
Reconstruction-Optimized Expert Pruning for Mixture-of-Experts Language Models
Jelena Markovic-Voronov, Petros Prastakos, Kayhan Behdin +7
Mixture-of-Experts (MoE) models reduce computation through sparse expert activation, but deployment still requires storing a large number of expert parameters. We study post-traini…
Maximum-modulus points on concentric circles
Yunpeng Li
We give a negative answer to the second question of Erdős Problem #1117: can the number of maximum-modulus points of a nonzero, non-monomial entire function on ten…
Smaller Models, Better Rejects: Preference Distillation Scaling
Rui Cai, Wenhui Zhu, Xiwen Chen +15
Preference distillation typically treats a teacher response as preferred and the student's own response as rejected. This assumes that self-generated failures are the most informat…
TREK: Distill to Explore, Reinforce to Refine
Yuanda Xu, Zhengze Zhou, Kayhan Behdin +10
Group Relative Policy Optimization (GRPO) is effective when the current policy already samples useful reasoning trajectories, but it stalls on hard prompts whose correct solution m…
Precision Tracked Transformer via Kalman Filtering, Kriging and Process Noise
Bo Long, Deepak Agarwal, Jelena Markovic-Voronov +2
The Transformer is the foundational building block of modern AI, yet offers no principled handling of \emph{uncertainty}, which is prevalent in real applications: cold-start tokens…
Sampling for Quality: Training-Free Reward-Guided LLM Decoding via Sequential Monte Carlo
Jelena Markovic-Voronov, Wenhui Zhu, Bo Long +5
We introduce a principled probabilistic framework for reward-guided decoding in large language models, addressing the limitations of standard decoding methods that optimize token-l…
Robust Batch-Level Query Routing for Large Language Models under Cost and Capacity Constraints
Jelena Markovic-Voronov, Kayhan Behdin, Yuanda Xu +3
We study the problem of routing queries to large language models (LLMs) under cost, GPU resources, and concurrency constraints. Prior per-query routing methods often fail to contro…
Black-box Selective Inference via Bootstrapping
Sifan Liu, Jelena Markovic-Voronov, Jonathan Taylor
Conditional selective inference requires an exact characterization of the selection event, which is often unavailable except for a few examples like the lasso. This work addresses…