7 papers
VaDar: Implicit Goal Conditioning via Value Disaggregation
Shashwat Saxena, Mehul Goel, Sreyas Venkataraman +2
Long-horizon tasks are important for modern AI and robotics, yet pose a challenge for reinforcement learning. Historically, a popular approach has been to decompose a task into sub…
Scalable Weak-to-Strong Verifier Hardening for Agent Environments
Chong Zeng, Ziqian Zhong, Kexun Zhang +5
Agent environments are essential for large language model (LLM) development. They use programmatic verifiers to turn complex outcomes into rewards. Verifiers can be attacked into a…
OGPO: Sample Efficient Full-Finetuning of Generative Control Policies
Sarvesh Patil, Mitsuhiko Nakamoto, Manan Agarwal +14
Generative control policies (GCPs), such as diffusion- and flow-based control policies, have emerged as effective parameterizations for robot learning. This work introduces Off-pol…
Hardening Agent Benchmarks with Adversarial Hacker-Fixer Loops
Ziqian Zhong, Ivgeni Segal, Ivan Bercovich +3
Agent benchmarks score submissions with outcome verifiers that are typically hand-written and brittle, leaving them open to reward hacking. We audit 1,968 tasks across five termina…
Terminal Wrench: A Dataset of 331 Reward-Hackable Environments and 3,632 Exploit Trajectories
Ivan Bercovich, Ivgeni Segal, Kexun Zhang +3
We release Terminal Wrench, a subset of 331 terminal-agent benchmark environments, copied from the popular open benchmarks that are demonstrably reward-hackable. The data set inclu…
Hodoscope: Unsupervised Monitoring for AI Misbehaviors
Ziqian Zhong, Shashwat Saxena, Aditi Raghunathan
Existing approaches to monitoring AI agents rely on supervised evaluation: human-written rules or LLM-based judges that check for known failure modes. However, novel misbehaviors m…
Jailbreaking in the Haystack
Rishi Rajesh Shah, Chen Henry Wu, Shashwat Saxena +3
Recent advances in long-context language models (LMs) have enabled million-token inputs, expanding their capabilities across complex tasks like computer-use agents. Yet, the safety…