Publications (6)
World Modeling with Probabilistic Structure Integration
Klemen Kotar, Wanhee Lee, Rahul Venkatesh +13
We present Probabilistic Structure Integration (PSI), a system for learning richly controllable and flexibly promptable world models from data. PSI consists of a three-step cycle.…
Self-Supervised Learning of Motion Concepts by Optimizing Counterfactuals
Stefan Stojanov, David Wendt, Seungwoo Kim +4
Estimating motion in videos is an essential computer vision problem with many downstream applications, including controllable video generation and robotics. Current solutions are p…
Unifying (Machine) Vision via Counterfactual World Modeling
Daniel M. Bear, Kevin Feigelis, Honglin Chen +5
Leading approaches in machine vision employ different architectures for different tasks, trained on costly task-specific labeled datasets. This complexity has held back progress in…
Understanding Physical Dynamics with Counterfactual World Modeling
Rahul Venkatesh, Honglin Chen, Kevin Feigelis +9
The ability to understand physical dynamics is critical for agents to act in the world. Here, we use Counterfactual World Modeling (CWM) to extract vision structures for dynamics u…
ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulation
Chuang Gan, Jeremy Schwartz, Seth Alter +21
We introduce ThreeDWorld (TDW), a platform for interactive multi-modal physical simulation. TDW enables simulation of high-fidelity sensory data and physical interactions between m…
Flexible and Efficient Long-Range Planning Through Curious Exploration
Aidan Curtis, Minjian Xin, Dilip Arumugam +2
Identifying algorithms that flexibly and efficiently discover temporally-extended multi-phase plans is an essential step for the advancement of robotics and model-based reinforceme…