37 citations · 65 across the 12 of their papers we have counts for
13 papers
WARM: On the Benefits of Weight Averaged Reward Models
Alexandre Ramé, Nino Vieillard, Léonard Hussenot +4
Aligning large language models (LLMs) with human preferences through reinforcement learning (RLHF) can lead to reward hacking, where LLMs exploit failures in the reward model (RM)…
On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes
Rishabh Agarwal, Nino Vieillard, Yongchao Zhou +4
Knowledge distillation (KD) is widely used for compressing a teacher model to reduce its inference cost and memory footprint, by training a smaller student model. However, current…
Factually Consistent Summarization via Reinforcement Learning with Textual Entailment Feedback
Paul Roit, Johan Ferret, Lior Shani +16
Despite the seeming success of contemporary grounded text generation systems, they often tend to generate factually inconsistent text with respect to their input. This phenomenon i…
Regularization and Variance-Weighted Regression Achieves Minimax Optimality in Linear MDPs: Theory and Practice
Toshinori Kitamura, Tadashi Kozuno, Yunhao Tang +12
Mirror descent value iteration (MDVI), an abstraction of Kullback-Leibler (KL) and entropy-regularized reinforcement learning (RL), has served as the basis for recent high-performi…
KL-Entropy-Regularized RL with a Generative Model is Minimax Optimal
Tadashi Kozuno, Wenhao Yang, Nino Vieillard +10
In this work, we consider and analyze the sample complexity of model-free reinforcement learning with a generative model. Particularly, we analyze mirror descent value iteration (M…
Implicitly Regularized RL with Implicit Q-Values
Nino Vieillard, Marcin Andrychowicz, Anton Raichuk +2
The -function is a central quantity in many Reinforcement Learning (RL) algorithms for which RL agents behave following a (soft)-greedy policy w.r.t. to . It is a powerful to…
Offline Reinforcement Learning as Anti-Exploration
Shideh Rezaeifar, Robert Dadashi, Nino Vieillard +4
Offline Reinforcement Learning (RL) aims at learning an optimal control from a fixed dataset, without interactions with the system. An agent in this setting should avoid selecting…
Offline Reinforcement Learning with Pseudometric Learning
Robert Dadashi, Shideh Rezaeifar, Nino Vieillard +3
Offline Reinforcement Learning methods seek to learn a policy from logged transitions of an environment, without any interaction. In the presence of function approximation, and und…
Munchausen Reinforcement Learning
Nino Vieillard, Olivier Pietquin, Matthieu Geist
Bootstrapping is a core mechanism in Reinforcement Learning (RL). Most algorithms, based on temporal differences, replace the true value of a transiting state by their current esti…
Leverage the Average: an Analysis of KL Regularization in RL
Nino Vieillard, Tadashi Kozuno, Bruno Scherrer +3
Recent Reinforcement Learning (RL) algorithms making use of Kullback-Leibler (KL) regularization as a core component have shown outstanding performance. Yet, only little is underst…
On Connections between Constrained Optimization and Reinforcement Learning
Nino Vieillard, Olivier Pietquin, Matthieu Geist
Dynamic Programming (DP) provides standard algorithms to solve Markov Decision Processes. However, these algorithms generally do not optimize a scalar objective function. In this p…
Momentum in Reinforcement Learning
Nino Vieillard, Bruno Scherrer, Olivier Pietquin +1
We adapt the optimization's concept of momentum to reinforcement learning. Seeing the state-action value functions as an analog to the gradients in optimization, we interpret momen…
Deep Conservative Policy Iteration
Nino Vieillard, Olivier Pietquin, Matthieu Geist
Conservative Policy Iteration (CPI) is a founding algorithm of Approximate Dynamic Programming (ADP). Its core principle is to stabilize greediness through stochastic mixtures of c…