10 papers
-0.6: Adaptive Streams of Thought in a Unified Vision-Language World Model
Khai Loong Aw, Klemen Kotar, Baihan Zhang +5
Humans reason about the physical world by flexibly combining different forms of inference as a problem unfolds. Achieving this flexibility is a major open challenge in AI: capabili…
Perceptual 3D Simulation With Physical World Modeling
Wanhee Lee, Klemen Kotar, Rahul Mysore Venkatesh +2
Predicting how a scene will evolve after a desired 3D transformation from images is a central goal in vision, graphics, and robotics. Yet unlike ideal simulators with full access t…
Physical Object Understanding with a Physically Controllable World Model
Rahul Venkatesh, Klemen Kotar, Lilian Naing Chen +9
A central challenge in visual intelligence is learning the physical structure of scenes from raw videos: how regions form objects and the laws that govern their interactions. Solvi…
Unified 3D Scene Understanding Through Physical World Modeling
Wanhee Lee, Klemen Kotar, Rahul Mysore Venkatesh +4
Understanding 3D scenes requires flexible combinations of visual reasoning tasks, including depth estimation, novel view synthesis, and object manipulation, all of which are essent…
Zero-shot World Models Are Developmentally Efficient Learners
Khai Loong Aw, Klemen Kotar, Wanhee Lee +6
Young children demonstrate early abilities to understand their physical world, estimating depth, motion, object coherence, interactions, and many other aspects of physical scene un…
World Modeling with Probabilistic Structure Integration
Klemen Kotar, Wanhee Lee, Rahul Venkatesh +13
We present Probabilistic Structure Integration (PSI), a system for learning richly controllable and flexibly promptable world models from data. PSI consists of a three-step cycle.…
Discovering and using Spelke segments
Rahul Venkatesh, Klemen Kotar, Lilian Naing Chen +10
Segments in computer vision are often defined by semantic considerations and are highly dependent on category-specific conventions. In contrast, developmental psychology suggests t…
3D Scene Understanding Through Local Random Access Sequence Modeling
Wanhee Lee, Klemen Kotar, Rahul Mysore Venkatesh +4
3D scene understanding from single images is a pivotal problem in computer vision with numerous downstream applications in graphics, augmented reality, and robotics. While diffusio…
Self-Supervised Learning of Motion Concepts by Optimizing Counterfactuals
Stefan Stojanov, David Wendt, Seungwoo Kim +4
Estimating motion in videos is an essential computer vision problem with many downstream applications, including controllable video generation and robotics. Current solutions are p…
Understanding Physical Dynamics with Counterfactual World Modeling
Rahul Venkatesh, Honglin Chen, Kevin Feigelis +9
The ability to understand physical dynamics is critical for agents to act in the world. Here, we use Counterfactual World Modeling (CWM) to extract vision structures for dynamics u…