5 papers
Integrating Simulation and Chain-of-thought Reasoning in Multimodal-Language Models For Physical Reasoning
YingQiao Wang, Eric Bigelow, Tomer Ullman +2
Multimodal language models often need to reason about the physical dynamics of the world in order to answer questions appropriately. Various methods have emerged for solving physic…
RoboFollow: Unveiling the Instruction Following Mirage in Embodied Agents
Chang Guo, Yukun Xie, Bohan Tan +6
Modern embodied agents achieve impressive success rates, yet their actual instruction-following ability is far weaker than these numbers suggest. We trace this illusion to a struct…
GASE: Gaussian Splatting-Based Automated System for Reconstructing Embodied-Simulation Environments
Jiawei Zhang, Yiming Yan, Chao Liang +8
Training embodied agents in the real world requires skilled operators and expensive hardware. Simulation environments offer a compelling alternative by enabling large-scale, cost-e…
Belief Dynamics Reveal the Dual Nature of In-Context Learning and Activation Steering
Eric Bigelow, Daniel Wurgaft, YingQiao Wang +4
Large language models (LLMs) can be controlled at inference time through prompts (in-context learning) and internal activations (activation steering). Different accounts have been…
Chain of Time: In-Context Physical Simulation with Image Generation Models
YingQiao Wang, Eric Bigelow, Boyi Li +1
We propose a novel cognitively-inspired prompting method to stabilize and interpret the physical simulation produced by autoregressive image generation models, called Chain-of-Time…