5 papers
TIPS: Turn-Level Information-Potential Reward Shaping for Search-Augmented LLMs
Yutao Xie, Nathaniel Thomas, Nicklas Hansen +3
Search-augmented large language models (LLMs) trained with reinforcement learning (RL) have achieved strong results on open-domain question answering (QA), but training still remai…
Rethinking Intermediate Representation for VLM-based Robot Manipulation
Weiliang Tang, Jialin Gao, Jia-Hui Pan +6
Vision-Language Model (VLM) is an important component to enable robust robot manipulation. Yet, using it to translate human instructions into an action-resolvable intermediate repr…
Incentivizing Multimodal Reasoning in Large Models for Direct Robot Manipulation
Weiliang Tang, Dong Jing, Jia-Hui Pan +5
Recent Large Multimodal Models have demonstrated remarkable reasoning capabilities, especially in solving complex mathematical problems and realizing accurate spatial perception. O…
Digi-Q: Learning Q-Value Functions for Training Device-Control Agents
Hao Bai, Yifei Zhou, Li Erran Li +2
While a number of existing approaches for building foundation model agents rely on prompting or fine-tuning with human demonstrations, it is not sufficient in dynamic environments…
GeoManip: Geometric Constraints as General Interfaces for Robot Manipulation
Weiliang Tang, Jia-Hui Pan, Yun-Hui Liu +4
We present GeoManip, a framework to enable generalist robots to leverage essential conditions derived from object and part relationships, as geometric constraints, for robot manipu…