11 papers · 1 filter
The Weakest Link: Distilling LLM Reasoning with Worst-Case Constrained Reinforcement Learning
Matthieu Zimmer, Xiaotong Ji, Tu Nguyen +1
Distilling the reasoning capabilities of large language models (LLMs) into smaller students is a central challenge for efficient deployment. Current approaches face a fundamental t…
Tail-Influence Sampling for CVaR Policy Evaluation
Pauline Bourigault, Xiaotong Ji, Matthieu Zimmer +2
Policies with similar mean returns can differ sharply in rare failures, yet estimating lower-tail conditional value-at-risk (CVaR) accurately can require many costly rollouts. When…
Beam Search as Test-Time Self-Distillation via Counterfactual Contexts
Su Ee Tan, Xiaotong Ji, Rasul Tutunov +2
Self-Distillation Fine-Tuning (SDFT) enables a language model to act as its own teacher: by conditioning on a demonstration, the model produces an implicit reward via pointwise mut…
Composable Decoding on the Probability Simplex: Theory and Implementation
Xiaotong Ji, Ahmed Khaled Khamis, Rasul Tutunov +2
Decoding for large language models is typically treated as a collection of isolated sampling strategies, with limited theoretical understanding of the behaviours they induce and ho…
iSDFT: Information-Proximal Self-Distillation for Continual Learning in LLMs
Ahmed Khaled Khamis, Xiaotong Ji, Hassan Jaber +4
On-policy self-distillation fine-tuning (SDFT) learns new skills from demonstrations while reducing forgetting, but it always distils toward the full demonstration-conditioned teac…
The -Combinator for LLMs: Solving Long-Context Rot with -Calculus
Amartya Roy, Rasul Tutunov, Xiaotong Ji +2
LLMs are increasingly used as general-purpose reasoners, but long inputs remain bottlenecked by a fixed context window. Recursive Language Models (RLMs) address this by externalisi…
Decoding as Optimisation on the Probability Simplex: From Top-K to Top-P (Nucleus) to Best-of-K Samplers
Xiaotong Ji, Rasul Tutunov, Matthieu Zimmer +1
Decoding sits between a language model and everything we do with it, yet it is still treated as a heuristic knob-tuning exercise. We argue decoding should be understood as a princi…
Scalable Power Sampling: Unlocking Efficient, Training-Free Reasoning for LLMs via Distribution Sharpening
Xiaotong Ji, Rasul Tutunov, Matthieu Zimmer +1
Reinforcement learning (RL) post-training is a dominant approach for improving the reasoning performance of large language models (LLMs), yet growing evidence suggests that its gai…
Rethinking Large Language Model Distillation: A Constrained Markov Decision Process Perspective
Matthieu Zimmer, Xiaotong Ji, Tu Nguyen +1
We introduce a novel approach to large language model (LLM) distillation by formulating it as a constrained reinforcement learning problem. While recent work has begun exploring th…
On Almost Surely Safe Alignment of Large Language Models at Inference-Time
Xiaotong Ji, Shyam Sundhar Ramesh, Matthieu Zimmer +3
We introduce a novel inference-time alignment approach for LLMs that aims to generate safe responses almost surely, i.e., with probability approaching one. Our approach models the…
Automatic Unit Test Data Generation and Actor-Critic Reinforcement Learning for Code Synthesis
Philip John Gorinski, Matthieu Zimmer, Gerasimos Lampouras +2
The advent of large pre-trained language models in the domain of Code Synthesis has shown remarkable performance on various benchmarks, treating the problem of Code Generation in a…