14 papers
Post-Training, Finetuning, and Steering:A Field Guide for Curating Training Data
Maria Attarian, Yifan Ruan, Claas Voelcker +5
While Vision-Language-Action models have enabled impressive cross-task transfer in zero-shot settings, \emph{adaptation} methods are often necessary to achieve consistent performan…
RH-Detect: A Unified Benchmark for Reward Hacking Detection
Junwei Quan, Evgenii Opryshko, Rohan Subramani +1
Reward hacking, where a model exploits an evaluation signal without completing the intended task, threatens the reliability of deployed language model systems. Existing datasets us…
World-Model Policy Arbiter for Goal-Conditioned Reinforcement Learning
Junwei Quan, Evgenii Opryshko, Nicholas Rhinehart +1
Offline goal-conditioned reinforcement learning (GCRL) has produced a diverse set of goal-reaching algorithms, yet no single algorithm performs best across environments, goals, and…
Modification-Considering Value Learning for Reward Hacking Mitigation in RL
Evgenii Opryshko, Umangi Jain, Igor Gilitschenski
Reinforcement learning agents can exploit misspecified reward signals to achieve high apparent returns while failing on the intended objective, a failure mode known as reward hacki…
QPILOTS: Efficient Test-Time Q-Steering for Flow Policies
Yifan Ruan, Chenyang Cao, Andreas Burger +7
Flow-matching and diffusion policies are expressive action generators, but optimizing them with temporal-difference reinforcement learning (RL) remains difficult. Effective policy…
Gamma-World: Generative Multi-Agent World Modeling Beyond Two Players
Fangfu Liu, Kai He, Tianchang Shen +7
World models for interactive video generation have largely focused on single-agent settings, where future observations are generated from a single control signal. However, many gen…
Dynamics Distillation for Efficient and Transferable Control Learning
Xunjiang Gu, Kashyap Chitta, Mahsa Golchoubian +2
Robust control policy learning for autonomous driving requires training environments to be both physically realistic and computationally scalable, properties that existing simulato…
Test-Time Graph Search for Goal-Conditioned Reinforcement Learning
Evgenii Opryshko, Junwei Quan, Claas Voelcker +2
Offline goal-conditioned reinforcement learning (GCRL) often struggles with long-horizon tasks, where errors in value estimation accumulate and produce unreliable policies. It is t…
Relative Entropy Pathwise Policy Optimization
Claas Voelcker, Axel Brunnbauer, Marcel Hussing +6
Score-function based methods for policy learning, such as REINFORCE and PPO, have delivered strong results in game-playing and robotics, yet their high variance often undermines tr…
Calibrated Value-Aware Model Learning with Probabilistic Environment Models
Claas Voelcker, Anastasiia Pedan, Arash Ahmadian +3
The idea of value-aware model learning, that models should produce accurate value estimates, has gained prominence in model-based reinforcement learning. The MuZero loss, which pen…
MAD-TD: Model-Augmented Data stabilizes High Update Ratio RL
Claas A Voelcker, Marcel Hussing, Eric Eaton +2
Building deep reinforcement learning (RL) agents that find a good policy with few samples has proven notoriously challenging. To achieve sample efficiency, recent work has explored…
When does Self-Prediction help? Understanding Auxiliary Tasks in Reinforcement Learning
Claas Voelcker, Tyler Kastner, Igor Gilitschenski +1
We investigate the impact of auxiliary learning tasks such as observation reconstruction and latent self-prediction on the representation learning problem in reinforcement learning…
Dissecting Deep RL with High Update Ratios: Combatting Value Divergence
Marcel Hussing, Claas Voelcker, Igor Gilitschenski +2
We show that deep reinforcement learning algorithms can retain their ability to learn without resetting network parameters in settings where the number of gradient updates greatly…
-models: Effective Decision-Aware Reinforcement Learning with Latent Models
Claas A Voelcker, Arash Ahmadian, Romina Abachi +2
The idea of decision-aware model learning, that models should be accurate where it matters for decision-making, has gained prominence in model-based reinforcement learning. While p…