Publications (26)
Decoding as Optimisation on the Probability Simplex: From Top-K to Top-P (Nucleus) to Best-of-K Samplers
Xiaotong Ji, Rasul Tutunov, Matthieu Zimmer +1
Decoding sits between a language model and everything we do with it, yet it is still treated as a heuristic knob-tuning exercise. We argue decoding should be understood as a princi…
Bourbaki: Self-Generated and Goal-Conditioned MDPs for Theorem Proving
Matthieu Zimmer, Xiaotong Ji, Rasul Tutunov +3
Reasoning remains a challenging task for large language models (LLMs), especially within the logically constrained environment of automated theorem proving (ATP), due to sparse rew…
Towards More Sample Efficiency in Reinforcement Learning with Data Augmentation
Yijiong Lin, Jiancong Huang, Matthieu Zimmer +2
Deep reinforcement learning (DRL) is a promising approach for adaptive robot control, but its current application to robotics is currently hindered by high sample requirements. We…
Rethinking Large Language Model Distillation: A Constrained Markov Decision Process Perspective
Matthieu Zimmer, Xiaotong Ji, Tu Nguyen +1
We introduce a novel approach to large language model (LLM) distillation by formulating it as a constrained reinforcement learning problem. While recent work has begun exploring th…
Tree-OPO: Off-policy Monte Carlo Tree-Guided Advantage Optimization for Multistep Reasoning
Bingning Huang, Tu Nguyen, Matthieu Zimmer
Recent advances in reasoning with large language models (LLMs) have shown the effectiveness of Monte Carlo Tree Search (MCTS) for generating high quality intermediate trajectories,…
Sample-Efficient Optimisation with Probabilistic Transformer Surrogates
Alexandre Maraval, Matthieu Zimmer, Antoine Grosnit +3
Faced with problems of increasing complexity, recent research in Bayesian Optimisation (BO) has focused on adapting deep probabilistic models as flexible alternatives to Gaussian P…
A Survey on Interpretable Reinforcement Learning
Claire Glanois, Paul Weng, Matthieu Zimmer +4
Although deep reinforcement learning has become a promising machine learning approach for sequential decision-making problems, it is still not mature enough for high-stake domains…
Multi-Task GRPO: Reliable LLM Reasoning Across Tasks
Shyam Sundhar Ramesh, Xiaotong Ji, Matthieu Zimmer +5
RL-based post-training with GRPO is widely used to improve large language models on individual reasoning tasks. However, real-world deployment requires reliable performance across…
Mixture of Attentions For Speculative Decoding
Matthieu Zimmer, Milan Gritta, Gerasimos Lampouras +2
The growth in the number of parameters of Large Language Models (LLMs) has led to a significant surge in computational requirements, making them challenging and costly to deploy. S…
Learning Fair Policies in Decentralized Cooperative Multi-Agent Reinforcement Learning
Matthieu Zimmer, Claire Glanois, Umer Siddique +1
We consider the problem of learning fair policies in (deep) cooperative multi-agent reinforcement learning (MARL). We formalize it in a principled way as the problem of optimizing…
Learning Fair Policies in Multiobjective (Deep) Reinforcement Learning with Average and Discounted Rewards
Umer Siddique, Paul Weng, Matthieu Zimmer
As the operations of autonomous systems generally affect simultaneously several users, it is crucial that their designs account for fairness considerations. In contrast to standard…
On Almost Surely Safe Alignment of Large Language Models at Inference-Time
Xiaotong Ji, Shyam Sundhar Ramesh, Matthieu Zimmer +3
We introduce a novel inference-time alignment approach for LLMs that aims to generate safe responses almost surely, i.e., with probability approaching one. Our approach models the…
Neuro-Symbolic Hierarchical Rule Induction
Claire Glanois, Xuening Feng, Zhaohui Jiang +4
We propose an efficient interpretable neuro-symbolic model to solve Inductive Logic Programming (ILP) problems. In this model, which is built from a set of meta-rules organised in…
The Model Knows, the Decoder Finds: Future Value Guided Particle Power Sampling
Tu Nguyen, Matthieu Zimmer, Rasul Tutunov +2
A recurring pattern in "reasoning without training" is that base LLMs already assign non-trivial probability mass to correct multi-step solutions; the bottleneck is locating these…
The -Combinator for LLMs: Solving Long-Context Rot with -Calculus
Amartya Roy, Rasul Tutunov, Xiaotong Ji +2
LLMs are increasingly used as general-purpose reasoners, but long inputs remain bottlenecked by a fixed context window. Recursive Language Models (RLMs) address this by externalisi…
End-to-End Meta-Bayesian Optimisation with Transformer Neural Processes
Alexandre Maraval, Matthieu Zimmer, Antoine Grosnit +1
Meta-Bayesian optimisation (meta-BO) aims to improve the sample efficiency of Bayesian optimisation by leveraging data from related tasks. While previous methods successfully meta-…
Invariant Transform Experience Replay: Data Augmentation for Deep Reinforcement Learning
Yijiong Lin, Jiancong Huang, Matthieu Zimmer +3
Deep Reinforcement Learning (RL) is a promising approach for adaptive robot control, but its current application to robotics is currently hindered by high sample requirements. To a…
Distilling Morphology-Conditioned Hypernetworks for Efficient Universal Morphology Control
Zheng Xiong, Risto Vuorio, Jacob Beck +3
Learning a universal policy across different robot morphologies can significantly improve learning efficiency and enable zero-shot generalization to unseen morphologies. However, l…
Differentiable Logic Machines
Matthieu Zimmer, Xuening Feng, Claire Glanois +6
The integration of reasoning, learning, and decision-making is key to build more general artificial intelligence systems. As a step in this direction, we propose a novel neural-log…
ROS-LLM: A ROS framework for embodied AI with task feedback and structured reasoning
Christopher E. Mower, Yuhui Wan, Hongzhan Yu +20
We present a framework for intuitive robot programming by non-experts, leveraging natural language prompts and contextual information from the Robot Operating System (ROS). Our sys…
Scalable Power Sampling: Unlocking Efficient, Training-Free Reasoning for LLMs via Distribution Sharpening
Xiaotong Ji, Rasul Tutunov, Matthieu Zimmer +1
Reinforcement learning (RL) post-training is a dominant approach for improving the reasoning performance of large language models (LLMs), yet growing evidence suggests that its gai…
Pangu-Agent: A Fine-Tunable Generalist Agent with Structured Reasoning
Filippos Christianos, Georgios Papoudakis, Matthieu Zimmer +13
A key method for creating Artificial Intelligence (AI) agents is Reinforcement Learning (RL). However, constructing a standalone RL policy that maps perception to action directly e…
Hyperparameter Auto-tuning in Self-Supervised Robotic Learning
Jiancong Huang, Juan Rojas, Matthieu Zimmer +3
Policy optimization in reinforcement learning requires the selection of numerous hyperparameters across different environments. Fixing them incorrectly may negatively impact optimi…
Risk-Controlled Lean-as-Judge for Natural-Language Mathematical Reasoning
Pauline Bourigault, Xiaotong Ji, Matthieu Zimmer +2
Lean is increasingly used to judge natural-language mathematical answers, but its signal is partial: many answers never formalize, and a failed proof may reflect an ill-typed state…
Automatic Unit Test Data Generation and Actor-Critic Reinforcement Learning for Code Synthesis
Philip John Gorinski, Matthieu Zimmer, Gerasimos Lampouras +2
The advent of large pre-trained language models in the domain of Code Synthesis has shown remarkable performance on various benchmarks, treating the problem of Code Generation in a…
Exploiting the Sign of the Advantage Function to Learn Deterministic Policies in Continuous Domains
Matthieu Zimmer, Paul Weng
In the context of learning deterministic policies in continuous domains, we revisit an approach, which was first proposed in Continuous Actor Critic Learning Automaton (CACLA) and…