39 citations · 61 across the 19 of their papers we have counts for
22 papers
Large Reward Models: Generalizable Online Robot Reward Generation with Vision-Language Models
Yanru Wu, Weiduo Yuan, Ang Qi +4
Reinforcement Learning (RL) has shown strong potential for improving robotic manipulation policies, yet its practical use remains bottlenecked by the difficulty of specifying rewar…
DreamPlan: Efficient Reinforcement Fine-Tuning of Vision-Language Planners via Video World Models
Emily Yue-Ting Jia, Weiduo Yuan, Tianheng Shi +3
Robotic manipulation requires sophisticated commonsense reasoning, a capability naturally possessed by large-scale Vision-Language Models (VLMs). While VLMs show promise as zero-sh…
: An Open Foundation Model Towards Universal Humanoid Loco-Manipulation
Songlin Wei, Hongyi Jing, Boqian Li +12
We introduce (Psi-Zero), an open foundation model to address challenging humanoid loco-manipulation tasks. While existing approaches often attempt to address this fundamental…
D-REX: Differentiable Real-to-Sim-to-Real Engine for Learning Dexterous Grasping
Haozhe Lou, Mingtong Zhang, Haoran Geng +9
Simulation provides a cost-effective and flexible platform for data generation and policy learning to develop robotic systems. However, bridging the gap between simulation and real…
FuzzingRL: Reinforcement Fuzz-Testing for Revealing VLM Failures
Jiajun Xu, Jiageng Mao, Ang Qi +5
Vision Language Models (VLMs) are prone to errors, and identifying where these errors occur is critical for ensuring the reliability and safety of AI systems. In this paper, we pro…
Seeing the Wind from a Falling Leaf
Zhiyuan Gao, Jiageng Mao, Hong-Xing Yu +5
A longstanding goal in computer vision is to model motions from videos, while the representations behind motions, i.e. the invisible physical interactions that cause objects to def…
Robot Learning from a Physical World Model
Jiageng Mao, Sicheng He, Hao-Ning Wu +9
We introduce PhysWorld, a framework that enables robot learning from video generation through physical world modeling. Recent video generation models can synthesize photorealistic…
ROVER: Benchmarking Reciprocal Cross-Modal Reasoning for Omnimodal Generation
Yongyuan Liang, Wei Chow, Feng Li +7
Unified multimodal models (UMMs) have emerged as a powerful paradigm for seamlessly unifying text and image understanding and generation. However, prevailing evaluations treat thes…
SeFA-Policy: Fast and Accurate Visuomotor Policy Learning with Selective Flow Alignment
Rong Xue, Jiageng Mao, Mingtong Zhang +1
Developing efficient and accurate visuomotor policies poses a central challenge in robotic imitation learning. While recent rectified flow approaches have advanced visuomotor polic…
Robot Learning from Any Images
Siheng Zhao, Jiageng Mao, Wei Chow +11
We introduce RoLA, a framework that transforms any in-the-wild image into an interactive, physics-enabled robotic environment. Unlike previous methods, RoLA operates directly on a…
InstantSfM: Towards GPU-Native SfM for the Deep Learning Era
Jiankun Zhong, Zitong Zhan, Quankai Gao +6
Structure-from-Motion (SfM) is a fundamental technique for recovering camera poses and scene structure from multi-view imagery, serving as a critical upstream component for applica…
Humanoid Everyday: A Comprehensive Robotic Dataset for Open-World Humanoid Manipulation
Zhenyu Zhao, Hongyi Jing, Xiawei Liu +7
From loco-motion to dextrous manipulation, humanoid robots have made remarkable strides in demonstrating complex full-body capabilities. However, the majority of current robot lear…
Learning an Implicit Physics Model for Image-based Fluid Simulation
Emily Yue-Ting Jia, Jiageng Mao, Zhiyuan Gao +2
Humans possess an exceptional ability to imagine 4D scenes, encompassing both motion and 3D geometry, from a single still image. This ability is rooted in our accumulated observati…
ManipBench: Benchmarking Vision-Language Models for Low-Level Robot Manipulation
Enyu Zhao, Vedant Raval, Hejia Zhang +5
Vision-Language Models (VLMs) have revolutionized artificial intelligence and robotics due to their commonsense reasoning capabilities. In robotic manipulation, VLMs are used prima…
RoboVerse: Towards a Unified Platform, Dataset and Benchmark for Scalable and Generalizable Robot Learning
Haoran Geng, Feishi Wang, Songlin Wei +34
Data scaling and standardized evaluation benchmarks have driven significant advances in natural language processing and computer vision. However, robotics faces unique challenges i…
PhysBench: Benchmarking and Enhancing Vision-Language Models for Physical World Understanding
Wei Chow, Jiageng Mao, Boyi Li +3
Understanding the physical world is a fundamental challenge in embodied AI, critical for enabling agents to perform complex tasks and operate safely in real-world environments. Whi…
DreamDrive: Generative 4D Scene Modeling from Street View Images
Jiageng Mao, Boyi Li, Boris Ivanovic +7
Synthesizing photo-realistic visual observations from an ego vehicle's driving trajectory is a critical step towards scalable training of self-driving models. Reconstruction-based…
Learning from Massive Human Videos for Universal Humanoid Pose Control
Jiageng Mao, Siheng Zhao, Siqi Song +7
Scalable learning of humanoid robots is crucial for their deployment in real-world applications. While traditional approaches primarily rely on reinforcement learning or teleoperat…
RAM: Retrieval-Based Affordance Transfer for Generalizable Zero-Shot Robotic Manipulation
Yuxuan Kuang, Junjie Ye, Haoran Geng +5
This work proposes a retrieve-and-transfer framework for zero-shot robotic manipulation, dubbed RAM, featuring generalizability across various objects, environments, and embodiment…
Driving Everywhere with Large Language Model Policy Adaptation
Boyi Li, Yue Wang, Jiageng Mao +4
Adapting driving behavior to new environments, customs, and laws is a long-standing problem in autonomous driving, precluding the widespread deployment of autonomous vehicles (AVs)…
A Language Agent for Autonomous Driving
Jiageng Mao, Junjie Ye, Yuxi Qian +2
Human-level driving is an ultimate goal of autonomous driving. Conventional approaches formulate autonomous driving as a perception-prediction-planning framework, yet their systems…
GPT-Driver: Learning to Drive with GPT
Jiageng Mao, Yuxi Qian, Junjie Ye +2
We present a simple yet effective approach that can transform the OpenAI GPT-3.5 model into a reliable motion planner for autonomous vehicles. Motion planning is a core challenge i…