1 citations · 1 across the 12 of their papers we have counts for
13 papers
WorldAgen: Unified State-Action Prediction with Test-Time World Model Training
Chi Wan, Kangrui Wang, Yuan Si +2
How can vision-language-action (VLA) models adapt to new environments where world dynamics shift? While recent research has combined world modeling and action prediction to improve…
MANIGUARD: A Benchmark and Data Suite for Specification-Grounded Safety Evaluation and Improvement of Robotic Manipulation
Yiyan Peng, Philip Wang, Simon Sinong Zhan +11
Foundation-model policies for robotic manipulation are advancing rapidly on task success, but rigorous evaluation of whether they succeed safely is still lacking. We introduce Mani…
Consistency-Driven Co-Evolution for Self-Supervised Cross-Representation Learning
Xuehang Guo, Pengyuan Li, Tom Hope +3
As chart images, tabular data, and visualization code play increasingly important roles across diverse domains, cross-representation understanding across these modalities poses fun…
CURV: Enhancing Chart Understanding Through Curriculum Visual Grounded Reasoning
Xuehang Guo, Pingyue Zhang, Ruiyi Zhang +6
Chart question answering (CQA) requires multimodal large language models (MLLMs) to integrate visual comprehension with logical reasoning, yet current models struggle with accurate…
HumanCLAW: Can Vision-Language Models Act Through a Body?
Li Siyao, Jiawei Gu, Shuai Liu +15
Evaluating whether a vision-language model (VLM) can act through a physical body is challenging. The outcome of an action couples the VLM's decision with motor control. When a task…
Long-Horizon Embodied Decision-Making via Multimodal Memory Compression
Bingxuan Li, Rui Yang, Cheng Qian +6
Agents are increasingly expected to act not only as task executors, but also as decision-makers on behalf of human users. This shift requires agents to accumulate evidence over lon…
SENTINEL: Failure-Driven Reinforcement Learning for Training Tool-Using Language Model Agents
Ziyi Wang, Yuxuan Lu, Yimeng Zhang +8
Language model agents are increasingly effective in solving realistic tasks through multi-turn tool use. However, training reliable tool-using agents remains challenging in practic…
Brick-Composer: Using MLLMs for Assembly with Diverse Bricks
Jiateng Liu, Bingxuan Li, Zhenhailong Wang +8
We dream of AI agents that can read arbitrary designs and construct real-world objects from reusable building blocks. As a first step toward this vision, we study whether multimoda…
DreamAvoid: Critical-Phase Test-Time Dreaming to Avoid Failures in VLA Policies
Xianzhe Fan, Yuxiang Lu, Shenyuan Gao +4
Vision-Language-Action (VLA) models are often brittle in fine-grained manipulation, where minor action errors during the critical phases can rapidly escalate into irrecoverable fai…
ODESteer: A Unified ODE-Based Steering Framework for LLM Alignment
Hongjue Zhao, Haosen Sun, Jiangtao Kong +8
Activation steering, or representation engineering, offers a lightweight approach to align large language models (LLMs) by manipulating their internal activations at inference time…
VAGEN: Reinforcing World Model Reasoning for Multi-Turn VLM Agents
Kangrui Wang, Pingyue Zhang, Zihan Wang +13
A key challenge in training Vision-Language Model (VLM) agents, compared to Language Model (LLM) agents, lies in the shift from textual states to complex visual observations. This…
Why Do LLM Agents Fail in Exploring New Environments? A World-Modeling Perspective
Shiqi Chen, Tongyao Zhu, Zian Wang +8
Large Language Models (LLMs) as agents often fail to improve in new environments. We identify and characterize a failure mode we call exploration collapse: under reinforcement lear…
SENTINEL: A Multi-Level Formal Framework for Safety Evaluation of Foundation Model-based Embodied Agents
Simon Sinong Zhan, Philip Wang, Yao Liu +13
We present SENTINEL, a framework for formally evaluating the physical safety of foundation model (FM)-based embodied agents. SENTINEL is the first to provide multi-level safety eva…