44 citations · 54 across the 18 of their papers we have counts for
19 papers
Perceptual 3D Simulation With Physical World Modeling
Wanhee Lee, Klemen Kotar, Rahul Mysore Venkatesh +2
Predicting how a scene will evolve after a desired 3D transformation from images is a central goal in vision, graphics, and robotics. Yet unlike ideal simulators with full access t…
Physical Object Understanding with a Physically Controllable World Model
Rahul Venkatesh, Klemen Kotar, Lilian Naing Chen +9
A central challenge in visual intelligence is learning the physical structure of scenes from raw videos: how regions form objects and the laws that govern their interactions. Solvi…
Unified 3D Scene Understanding Through Physical World Modeling
Wanhee Lee, Klemen Kotar, Rahul Mysore Venkatesh +4
Understanding 3D scenes requires flexible combinations of visual reasoning tasks, including depth estimation, novel view synthesis, and object manipulation, all of which are essent…
Zero-shot World Models Are Developmentally Efficient Learners
Khai Loong Aw, Klemen Kotar, Wanhee Lee +6
Young children demonstrate early abilities to understand their physical world, estimating depth, motion, object coherence, interactions, and many other aspects of physical scene un…
World Modeling with Probabilistic Structure Integration
Klemen Kotar, Wanhee Lee, Rahul Venkatesh +13
We present Probabilistic Structure Integration (PSI), a system for learning richly controllable and flexibly promptable world models from data. PSI consists of a three-step cycle.…
Representing Speech Through Autoregressive Prediction of Cochlear Tokens
Greta Tuckute, Klemen Kotar, Evelina Fedorenko +1
We introduce AuriStream, a biologically inspired model for encoding speech via a two-stage framework inspired by the human auditory processing hierarchy. The first stage transforms…
Discovering and using Spelke segments
Rahul Venkatesh, Klemen Kotar, Lilian Naing Chen +10
Segments in computer vision are often defined by semantic considerations and are highly dependent on category-specific conventions. In contrast, developmental psychology suggests t…
Taming generative video models for zero-shot optical flow extraction
Seungwoo Kim, Khai Loong Aw, Klemen Kotar +8
Extracting optical flow from videos remains a core computer vision problem. Motivated by the recent success of large general-purpose models, we ask whether frozen self-supervised v…
Model Connectomes: A Generational Approach to Data-Efficient Language Models
Klemen Kotar, Greta Tuckute
Biological neural networks are shaped both by evolution across generations and by individual learning within an organism's lifetime, whereas standard artificial neural networks und…
3D Scene Understanding Through Local Random Access Sequence Modeling
Wanhee Lee, Klemen Kotar, Rahul Mysore Venkatesh +4
3D scene understanding from single images is a pivotal problem in computer vision with numerous downstream applications in graphics, augmented reality, and robotics. While diffusio…
WhisBERT: Multimodal Text-Audio Language Modeling on 100M Words
Lukas Wolf, Greta Tuckute, Klemen Kotar +4
Training on multiple modalities of input can augment the capabilities of a language model. Here, we ask whether such a training regime can improve the quality and efficiency of the…
Understanding Physical Dynamics with Counterfactual World Modeling
Rahul Venkatesh, Honglin Chen, Kevin Feigelis +9
The ability to understand physical dynamics is critical for agents to act in the world. Here, we use Counterfactual World Modeling (CWM) to extract vision structures for dynamics u…
Are These the Same Apple? Comparing Images Based on Object Intrinsics
Klemen Kotar, Stephen Tian, Hong-Xing Yu +2
The human visual system can effortlessly recognize an object under different extrinsic factors such as lighting, object poses, and background, yet current computer vision systems o…
Unifying (Machine) Vision via Counterfactual World Modeling
Daniel M. Bear, Kevin Feigelis, Honglin Chen +5
Leading approaches in machine vision employ different architectures for different tasks, trained on costly task-specific labeled datasets. This complexity has held back progress in…
ENTL: Embodied Navigation Trajectory Learner
Klemen Kotar, Aaron Walsman, Roozbeh Mottaghi
We propose Embodied Navigation Trajectory Learner (ENTL), a method for extracting long sequence representations for embodied navigation. Our approach unifies world modeling, locali…
Break and Make: Interactive Structural Understanding Using LEGO Bricks
Aaron Walsman, Muru Zhang, Klemen Kotar +3
Visual understanding of geometric structures with complex spatial relationships is a fundamental component of human intelligence. As children, we learn how to reason about structur…
Interactron: Embodied Adaptive Object Detection
Klemen Kotar, Roozbeh Mottaghi
Over the years various methods have been proposed for the problem of object detection. Recently, we have witnessed great strides in this domain owing to the emergence of powerful d…
Contrasting Contrastive Self-Supervised Representation Learning Pipelines
Klemen Kotar, Gabriel Ilharco, Ludwig Schmidt +2
In the past few years, we have witnessed remarkable breakthroughs in self-supervised representation learning. Despite the success and adoption of representations learned through th…
AllenAct: A Framework for Embodied AI Research
Luca Weihs, Jordi Salvador, Klemen Kotar +4
The domain of Embodied AI, in which agents learn to complete tasks through interaction with their environment from egocentric observations, has experienced substantial growth with…