112 citations · 304 across the 149 of their papers we have counts for
33 papers · 1 filter
Streaming Detection of Queried Event Start
Cristobal Eyzaguirre, Eric Tang, Shyamal Buch +3
Robotics, autonomous driving, augmented reality, and many embodied computer vision applications must quickly react to user-defined events unfolding in real time. We address this se…
ZeroHSI: Zero-Shot 4D Human-Scene Interaction by Video Generation
Hongjie Li, Hong-Xing Yu, Jiaman Li +1
Human-scene interaction (HSI) generation is crucial for applications in embodied AI, virtual reality, and robotics. Yet, existing methods cannot synthesize interactions in unseen e…
Birth and Death of a Rose
Chen Geng, Yunzhi Zhang, Shangzhe Wu +1
We study the problem of generating temporal object intrinsics -- temporally evolving sequences of object geometry, reflectance, and texture, such as a blooming rose -- from pre-tra…
CRAFT: Designing Creative and Functional 3D Objects
Michelle Guo, Mia Tang, Hannah Cha +3
For designing a wide range of everyday objects, the design process should be aware of both the human body and the underlying semantics of the design specification. However, these t…
LayoutVLM: Differentiable Optimization of 3D Layout via Vision-Language Models
Fan-Yun Sun, Weiyu Liu, Siyi Gu +6
Spatial reasoning is a fundamental aspect of human cognition, enabling intuitive understanding and manipulation of objects in three-dimensional space. While foundation models demon…
Diffusion Self-Distillation for Zero-Shot Customized Image Generation
Shengqu Cai, Eric Chan, Yunzhi Zhang +3
Text-to-image diffusion models produce impressive results but are frustrating tools for artists who desire fine-grained control. For example, a common use case is to create images…
IKEA Manuals at Work: 4D Grounding of Assembly Instructions on Internet Videos
Yunong Liu, Cristobal Eyzaguirre, Manling Li +6
Shape assembly is a ubiquitous task in daily life, integral for constructing complex 3D structures like IKEA furniture. While significant progress has been made in developing auton…
HourVideo: 1-Hour Video-Language Understanding
Keshigeyan Chandrasegaran, Agrim Gupta, Lea M. Hadzic +7
We present HourVideo, a benchmark dataset for hour-long video-language understanding. Our dataset consists of a novel task suite comprising summarization, perception (recall, track…
Lifting Motion to the 3D World via 2D Diffusion
Jiaman Li, C. Karen Liu, Jiajun Wu
Estimating 3D motion from 2D observations is a long-standing research challenge. Prior work typically requires training on datasets containing ground truth 3D motions, limiting the…
Learning Smooth Humanoid Locomotion through Lipschitz-Constrained Policies
Zixuan Chen, Xialin He, Yen-Jen Wang +8
Reinforcement learning combined with sim-to-real transfer offers a general framework for developing locomotion controllers for legged robots. To facilitate successful deployment in…
Automated Creation of Digital Cousins for Robust Policy Learning
Tianyuan Dai, Josiah Wong, Yunfan Jiang +5
Training robot policies in the real world can be unsafe, costly, and difficult to scale. Simulation serves as an inexpensive and potentially limitless source of training data, but…
MARPLE: A Benchmark for Long-Horizon Inference
Emily Jin, Zhuoyi Huang, Jan-Philipp Fränken +7
Reconstructing past events requires reasoning across long time horizons. To figure out what happened, we need to use our prior knowledge about the world and human behavior and draw…
The Scene Language: Representing Scenes with Programs, Words, and Embeddings
Yunzhi Zhang, Zizhang Li, Matt Zhou +2
We introduce the Scene Language, a visual scene representation that concisely and precisely describes the structure, semantics, and identity of visual scenes. It represents a scene…
Generalizable Humanoid Manipulation with 3D Diffusion Policies
Yanjie Ze, Zixuan Chen, Wenhao Wang +5
Humanoid robots capable of autonomous operation in diverse environments have long been a goal for roboticists. However, autonomous manipulation by humanoid robots has largely been…
Embodied Agent Interface: Benchmarking LLMs for Embodied Decision Making
Manling Li, Shiyu Zhao, Qineng Wang +12
We aim to evaluate Large Language Models (LLMs) for embodied decision making. While a significant body of work has been leveraging LLMs for decision making in embodied environments…
DiffSound: Differentiable Modal Sound Rendering and Inverse Rendering for Diverse Inference Tasks
Xutong Jin, Chenxi Xu, Ruohan Gao +3
Accurately estimating and simulating the physical properties of objects from real-world sound recordings is of great practical importance in the fields of vision, graphics, and rob…
An Eulerian Vortex Method on Flow Maps
Sinan Wang, Yitong Deng, Molin Deng +6
We present an Eulerian vortex method based on the theory of flow maps to simulate the complex vortical motions of incompressible fluids. Central to our method is the novel incorpor…
FactorSim: Generative Simulation via Factorized Representation
Fan-Yun Sun, S. I. Harini, Angela Yi +6
Generating simulations to train intelligent agents in game-playing and robotics from natural language input, from user input or task documentation, remains an open-ended challenge.…
What Makes a Maze Look Like a Maze?
Joy Hsu, Jiayuan Mao, Joshua B. Tenenbaum +2
A unique aspect of human visual understanding is the ability to flexibly interpret abstract concepts: acquiring lifted rules explaining what they symbolize, grounding them across f…
View-Invariant Policy Learning via Zero-Shot Novel View Synthesis
Stephen Tian, Blake Wulfe, Kyle Sargent +4
Large-scale visuomotor policy learning is a promising approach toward developing generalizable manipulation systems. Yet, policies that can be deployed on diverse embodiments, envi…
D5RL: Diverse Datasets for Data-Driven Deep Reinforcement Learning
Rafael Rafailov, Kyle Hatch, Anikait Singh +9
Offline reinforcement learning algorithms hold the promise of enabling data-driven RL methods that do not require costly or dangerous real-world exploration and benefit from large…
Enhancing Equitable Access to AI in Housing and Homelessness System of Care through Federated Learning
Musa Taib, Jiajun Wu, Steve Drew +1
The top priority of a Housing and Homelessness System of Care (HHSC) is to connect people experiencing homelessness to supportive housing. An HHSC typically consists of many agenci…
RoboPack: Learning Tactile-Informed Dynamics Models for Dense Packing
Bo Ai, Stephen Tian, Haochen Shi +4
Tactile feedback is critical for understanding the dynamics of both rigid and deformable objects in many manipulation tasks, such as non-prehensile manipulation and dense packing.…
WonderWorld: Interactive 3D Scene Generation from a Single Image
Hong-Xing Yu, Haoyi Duan, Charles Herrmann +2
We present WonderWorld, a novel framework for interactive 3D scene generation that enables users to interactively specify scene contents and layout and see the created scenes in lo…