9 papers
F-GRPO: Don't Let Your Policy Learn the Obvious and Forget the Rare
Daniil Plyusov, Alexey Gorbatovski, Boris Shaposhnikov +4
Reinforcement Learning with Verifiable Rewards (RLVR) is commonly based on group sampling to estimate advantages and stabilize policy updates. In practice, computational limits oft…
The Differences Between Direct Alignment Algorithms are a Blur
Alexey Gorbatovski, Boris Shaposhnikov, Viacheslav Sinii +2
Direct Alignment Algorithms (DAAs) simplify LLM alignment by directly optimizing policies, bypassing reward modeling and RL. While DAAs differ in their use of SFT (one-stage vs. tw…
Small Vectors, Big Effects: A Mechanistic Study of RL-Induced Reasoning via Steering Vectors
Viacheslav Sinii, Nikita Balagansky, Gleb Gerasimov +6
The mechanisms by which reasoning training reshapes LLMs' internal computations remain unclear. We study lightweight steering vectors inserted into the base model's residual stream…
ESSA: Evolutionary Strategies for Scalable Alignment
Daria Korotyshova, Boris Shaposhnikov, Alexey Malakhov +7
Alignment of Large Language Models (LLMs) typically relies on Reinforcement Learning from Human Feedback (RLHF) with gradient-based optimizers such as Proximal Policy Optimization…
Steering LLM Reasoning Through Bias-Only Adaptation
Viacheslav Sinii, Alexey Gorbatovski, Artem Cherepanov +3
We show that training a single -dimensional steering vector per layer with reinforcement learning, while freezing all base weights, matches the accuracy of fully RL-tuned reason…
Enhancing Vision-Language Model Training with Reinforcement Learning in Synthetic Worlds for Real-World Success
George Bredis, Stanislav Dereka, Viacheslav Sinii +2
Interactive multimodal agents must convert raw visual observations into coherent sequences of language-conditioned actions -- a capability that current vision-language models (VLMs…