15 papers
Safety Hacking in Constrained Best-of- Inference-time Scaling
Akifumi Wachi, Takumi Tanabe, Youhei Akimoto
Inference-time pipelines often sample multiple outputs, filter them with a learned safety model, and return the proxy-feasible output with the highest learned reward. We show that…
Sample-Efficient Hypergradient Estimation for Decentralized Bi-Level Reinforcement Learning
Mikoto Kudo, Takumi Tanabe, Akifumi Wachi +1
Many strategic decision-making problems, such as environment design for warehouse robots, can be naturally formulated as bi-level reinforcement learning (RL), where a leader agent…
Interaction-Limited Safe Continuous-Time RL for Dynamical Medical Treatment
Xun Shen, Yuepeng Wang, Akifumi Wachi +13
Dynamic medical treatment requires deciding treatment intensity and intervention timing, while patient states evolve continuously and adverse events may occur between clinical inte…
MedGym:A Unified Continuous-Time Benchmark for Dynamic Medical Treatment Reinforcement Learning
Yuepeng Wang, Ken Kawano, Yongqi Zhou +11
Medical treatment recommendation poses several challenges to reinforcement learning (RL): patient physiology evolves in continuous time, measurements and interventions are performe…
How Neural Reward Models Learn Features for Policy Optimization: A Single-Index Analysis
Rei Higuchi, Ryotaro Kawata, Akifumi Wachi +3
Reward modeling is not only a prediction problem: in KL-regularized policy optimization, the learned reward is exponentiated to define the deployed policy, so downstream value depe…
Inference-Aware Meta-Alignment of LLMs via Non-Linear GRPO
Shokichi Takakura, Akifumi Wachi, Rei Higuchi +2
Aligning large language models (LLMs) to diverse human preferences is fundamentally challenging since criteria can often conflict with each other. Inference-time alignment methods…