10 papers
HumanCLAW: Can Vision-Language Models Act Through a Body?
Li Siyao, Jiawei Gu, Shuai Liu +15
Evaluating whether a vision-language model (VLM) can act through a physical body is challenging. The outcome of an action couples the VLM's decision with motor control. When a task…
EgoExoMoCap: Distributed Ego-Exo Human Motion Capture
Jiaxi Jiang, Bharat Lal Bhatnagar, Nan Yang +6
Human motion capture from head-mounted devices (HMDs) offers a scalable way to acquire real-world human motion and interaction data, which is crucial for applications in embodied A…
EC: Video Generation with 3D Environmental Memory and Ego-Exo Human Pose Control
Qiao Gu, Lingni Ma, Adam W Harley +3
Controllable and physically grounded egocentric video generation is essential for embodied agents to reason about how their own and others' actions manifest and change the world. C…
LAMP: Localization Aware Multi-camera People Tracking in Metric 3D World
Nan Yang, Julian Straub, Fan Zhang +3
Tracking 3D human motion from egocentric multi-camera headset is challenged by severe egomotion, partial visibility or occlusions and lack of training data. Existing methods design…
Boxer: Robust Lifting of Open-World 2D Bounding Boxes to 3D
Daniel DeTone, Tianwei Shen, Fan Zhang +4
Detecting and localizing objects in space is a fundamental computer vision problem. While much progress has been made to solve 2D object detection, 3D object localization is much l…
NymeriaPlus: Enriching Nymeria Dataset with Additional Annotations and Data
Daniel DeTone, Federica Bogo, Eric-Tuan Le +7
The Nymeria Dataset, released in 2024, is a large-scale collection of in-the-wild human activities captured with multiple egocentric wearable devices that are spatially localized a…
EgoLM: Multi-Modal Language Model of Egocentric Motions
Fangzhou Hong, Vladimir Guzov, Hyo Jin Kim +4
As the prevalence of wearable devices, learning egocentric motions becomes essential to develop contextual AI. In this work, we present EgoLM, a versatile framework that tracks and…
HMD^2: Environment-aware Motion Generation from Single Egocentric Head-Mounted Device
Vladimir Guzov, Yifeng Jiang, Fangzhou Hong +5
This paper investigates the generation of realistic full-body human motion using a single head-mounted device with an outward-facing color camera and the ability to perform visual…
Nymeria: A Massive Collection of Multimodal Egocentric Daily Motion in the Wild
Lingni Ma, Yuting Ye, Fangzhou Hong +14
We introduce Nymeria - a large-scale, diverse, richly annotated human motion dataset collected in the wild with multiple multimodal egocentric devices. The dataset comes with a) fu…
DivaTrack: Diverse Bodies and Motions from Acceleration-Enhanced Three-Point Trackers
Dongseok Yang, Jiho Kang, Lingni Ma +3
Full-body avatar presence is crucial for immersive social and environmental interactions in digital reality. However, current devices only provide three six degrees of freedom (DOF…