8 papers · 1 filter
Efficient RL Training for LLMs with Experience Replay
Charles Arnal, Vivien Cabannes, Taco Cohen +2
While Experience Replay - the practice of storing rollouts and reusing them multiple times during training - is a foundational technique in general RL, it remains largely unexplore…
Provable Benefits of In-Tool Learning for Large Language Models
Sam Houliston, Ambroise Odonnat, Charles Arnal +1
Tool-augmented language models, equipped with retrieval, memory, or external APIs, are reshaping AI, yet their theoretical advantages remain underexplored. In this paper, we addres…
Asymmetric REINFORCE for off-Policy Reinforcement Learning: Balancing positive and negative rewards
Charles Arnal, Gaëtan Narozniak, Vivien Cabannes +3
Reinforcement learning (RL) is increasingly used to align large language models (LLMs). Off-policy methods offer greater implementation simplicity and data efficiency than on-polic…
Easing Optimization Paths: a Circuit Perspective
Ambroise Odonnat, Wassim Bouaziz, Vivien Cabannes
Gradient descent is the method of choice for training large artificial intelligence systems. As these systems become larger, a better understanding of the mechanisms behind gradien…
A Mechanistic Study of Transformers Training Dynamics
Ambroise Odonnat, Wassim Bouaziz, Vivien Cabannes
Large-scale pretraining of transformers has been central to the success of foundation models. However, the scale of those models limits our understanding of the mechanisms at play…
Iteration Head: A Mechanistic Study of Chain-of-Thought
Vivien Cabannes, Charles Arnal, Wassim Bouaziz +3
Chain-of-Thought (CoT) reasoning is known to improve Large Language Models both empirically and in terms of theoretical approximation power. However, our understanding of the inner…