5 papers
MM-ReCoder: Advancing Chart-to-Code Generation with Reinforcement Learning and Self-Correction
Zitian Tang, Xu Zhang, Jianbo Yuan +4
Multimodal Large Language Models (MLLMs) have recently demonstrated promising capabilities in multimodal coding tasks such as chart-to-code generation. However, existing methods pr…
Goal Force: Teaching Video Models To Accomplish Physics-Conditioned Goals
Nate Gillman, Yinghua Zhou, Zitian Tang +6
Recent advancements in video generation have enabled the development of ``world models'' capable of simulating potential futures for robotics and planning. However, specifying prec…
How Can Objects Help Video-Language Understanding?
Zitian Tang, Shijie Wang, Junho Cho +2
Do we still need to represent objects explicitly in multimodal large language models (MLLMs)? To one extreme, pre-trained encoders convert images into visual tokens, with which obj…
Spacewalk-18: A Benchmark for Multimodal and Long-form Procedural Video Understanding in Novel Domains
Zitian Tang, Rohan Myer Krishnan, Zhiqiu Yu +1
Learning from (procedural) videos has increasingly served as a pathway for embodied agents to acquire skills from human demonstrations. To do this, video understanding models must…
What Happened 3 Seconds Ago? Inferring the Past with Thermal Imaging
Zitian Tang, Wenjie Ye, Wei-Chiu Ma +1
Inferring past human motion from RGB images is challenging due to the inherent uncertainty of the prediction problem. Thermal images, on the other hand, encode traces of past human…