6 papers
Cosmos 3: Omnimodal World Models for Physical AI
NVIDIA, :, Aditi +293
We introduce Cosmos 3, a family of omnimodal world models designed to jointly process and generate language, image, video, audio, and action sequences within a unified mixture-of-t…
Attend Before Attention: Efficient and Scalable Video Understanding via Autoregressive Gazing
Baifeng Shi, Stephanie Fu, Long Lian +10
Multi-modal large language models (MLLMs) have advanced general-purpose video understanding but struggle with long, high-resolution videos -- they process every pixel equally in th…
The Sound of Simulation: Learning Multimodal Sim-to-Real Robot Policies with Generative Audio
Renhao Wang, Haoran Geng, Tingle Li +7
Robots must integrate multiple sensory modalities to act effectively in the real world. Yet, learning such multimodal policies at scale remains challenging. Simulation offers a via…
Wolf: Dense Video Captioning with a World Summarization Framework
Boyi Li, Ligeng Zhu, Ran Tian +20
We propose Wolf, a WOrLd summarization Framework for accurate video captioning. Wolf is an automated captioning framework that adopts a mixture-of-experts approach, leveraging comp…
Interactive Task Planning with Language Models
Boyi Li, Philipp Wu, Pieter Abbeel +1
An interactive robot framework accomplishes long-horizon task planning and can easily generalize to new goals and distinct tasks, even during execution. However, most traditional m…
Synthesizing Moving People with 3D Control
Boyi Li, Junming Chen, Jathushan Rajasegaran +3
In this paper, we present a diffusion model-based framework for animating people from a single image for a given target 3D motion sequence. Our approach has two core components: a)…