18 papers
A Safe Action Is Not Enough: Feasible-Future Decoding for Vision-Language-Action Policies
Tu Nguyen, Matthieu Zimmer, Vu Anh Vu +4
A safe action is not necessarily a viable one. A frozen vision-language-action (VLA) policy can favor a locally admissible move that leaves no policy-supported route to safe task c…
The Weakest Link: Distilling LLM Reasoning with Worst-Case Constrained Reinforcement Learning
Matthieu Zimmer, Xiaotong Ji, Tu Nguyen +1
Distilling the reasoning capabilities of large language models (LLMs) into smaller students is a central challenge for efficient deployment. Current approaches face a fundamental t…
Tail-Influence Sampling for CVaR Policy Evaluation
Pauline Bourigault, Xiaotong Ji, Matthieu Zimmer +2
Policies with similar mean returns can differ sharply in rare failures, yet estimating lower-tail conditional value-at-risk (CVaR) accurately can require many costly rollouts. When…
Beam Search as Test-Time Self-Distillation via Counterfactual Contexts
Su Ee Tan, Xiaotong Ji, Rasul Tutunov +2
Self-Distillation Fine-Tuning (SDFT) enables a language model to act as its own teacher: by conditioning on a demonstration, the model produces an implicit reward via pointwise mut…
Composable Decoding on the Probability Simplex: Theory and Implementation
Xiaotong Ji, Ahmed Khaled Khamis, Rasul Tutunov +2
Decoding for large language models is typically treated as a collection of isolated sampling strategies, with limited theoretical understanding of the behaviours they induce and ho…
iSDFT: Information-Proximal Self-Distillation for Continual Learning in LLMs
Ahmed Khaled Khamis, Xiaotong Ji, Hassan Jaber +4
On-policy self-distillation fine-tuning (SDFT) learns new skills from demonstrations while reducing forgetting, but it always distils toward the full demonstration-conditioned teac…
The Model Knows, the Decoder Finds: Future Value Guided Particle Power Sampling
Tu Nguyen, Matthieu Zimmer, Rasul Tutunov +2
A recurring pattern in "reasoning without training" is that base LLMs already assign non-trivial probability mass to correct multi-step solutions; the bottleneck is locating these…
Risk-Controlled Lean-as-Judge for Natural-Language Mathematical Reasoning
Pauline Bourigault, Xiaotong Ji, Matthieu Zimmer +2
Lean is increasingly used to judge natural-language mathematical answers, but its signal is partial: many answers never formalize, and a failed proof may reflect an ill-typed state…
The -Combinator for LLMs: Solving Long-Context Rot with -Calculus
Amartya Roy, Rasul Tutunov, Xiaotong Ji +2
LLMs are increasingly used as general-purpose reasoners, but long inputs remain bottlenecked by a fixed context window. Recursive Language Models (RLMs) address this by externalisi…
Decoding as Optimisation on the Probability Simplex: From Top-K to Top-P (Nucleus) to Best-of-K Samplers
Xiaotong Ji, Rasul Tutunov, Matthieu Zimmer +1
Decoding sits between a language model and everything we do with it, yet it is still treated as a heuristic knob-tuning exercise. We argue decoding should be understood as a princi…
Multi-Task GRPO: Reliable LLM Reasoning Across Tasks
Shyam Sundhar Ramesh, Xiaotong Ji, Matthieu Zimmer +5
RL-based post-training with GRPO is widely used to improve large language models on individual reasoning tasks. However, real-world deployment requires reliable performance across…
Scalable Power Sampling: Unlocking Efficient, Training-Free Reasoning for LLMs via Distribution Sharpening
Xiaotong Ji, Rasul Tutunov, Matthieu Zimmer +1
Reinforcement learning (RL) post-training is a dominant approach for improving the reasoning performance of large language models (LLMs), yet growing evidence suggests that its gai…
Rethinking Large Language Model Distillation: A Constrained Markov Decision Process Perspective
Matthieu Zimmer, Xiaotong Ji, Tu Nguyen +1
We introduce a novel approach to large language model (LLM) distillation by formulating it as a constrained reinforcement learning problem. While recent work has begun exploring th…
Tree-OPO: Off-policy Monte Carlo Tree-Guided Advantage Optimization for Multistep Reasoning
Bingning Huang, Tu Nguyen, Matthieu Zimmer
Recent advances in reasoning with large language models (LLMs) have shown the effectiveness of Monte Carlo Tree Search (MCTS) for generating high quality intermediate trajectories,…
Bourbaki: Self-Generated and Goal-Conditioned MDPs for Theorem Proving
Matthieu Zimmer, Xiaotong Ji, Rasul Tutunov +3
Reasoning remains a challenging task for large language models (LLMs), especially within the logically constrained environment of automated theorem proving (ATP), due to sparse rew…
On Almost Surely Safe Alignment of Large Language Models at Inference-Time
Xiaotong Ji, Shyam Sundhar Ramesh, Matthieu Zimmer +3
We introduce a novel inference-time alignment approach for LLMs that aims to generate safe responses almost surely, i.e., with probability approaching one. Our approach models the…
Mixture of Attentions For Speculative Decoding
Matthieu Zimmer, Milan Gritta, Gerasimos Lampouras +2
The growth in the number of parameters of Large Language Models (LLMs) has led to a significant surge in computational requirements, making them challenging and costly to deploy. S…
Automatic Unit Test Data Generation and Actor-Critic Reinforcement Learning for Code Synthesis
Philip John Gorinski, Matthieu Zimmer, Gerasimos Lampouras +2
The advent of large pre-trained language models in the domain of Code Synthesis has shown remarkable performance on various benchmarks, treating the problem of Code Generation in a…