38 citations · 156 across the 18 of their papers we have counts for
18 papers
Harmony4D: A Video Dataset for In-The-Wild Close Human Interactions
Rawal Khirodkar, Jyun-Ting Song, Jinkun Cao +2
Understanding how humans interact with each other is key to building realistic multi-human virtual reality systems. This area remains relatively unexplored due to the lack of large…
Omnigrasp: Grasping Diverse Objects with Simulated Humanoids
Zhengyi Luo, Jinkun Cao, Sammy Christen +3
We present a method for controlling a simulated humanoid to grasp an object and move it to follow an object's trajectory. Due to the challenges in controlling a humanoid with dexte…
OmniH2O: Universal and Dexterous Human-to-Humanoid Whole-Body Teleoperation and Learning
Tairan He, Zhengyi Luo, Xialin He +6
We present OmniH2O (Omni Human-to-Humanoid), a learning-based system for whole-body humanoid teleoperation and autonomy. Using kinematic pose as a universal control interface, Omni…
Real-Time Simulated Avatar from Head-Mounted Sensors
Zhengyi Luo, Jinkun Cao, Rawal Khirodkar +4
We present SimXR, a method for controlling a simulated avatar from information (headset pose and cameras) obtained from AR / VR headsets. Due to the challenging viewpoint of head-m…
Learning Human-to-Humanoid Real-Time Whole-Body Teleoperation
Tairan He, Zhengyi Luo, Wenli Xiao +4
We present Human to Humanoid (H2O), a reinforcement learning (RL) based framework that enables real-time whole-body teleoperation of a full-sized humanoid robot with only an RGB ca…
Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives
Kristen Grauman, Andrew Westbury, Lorenzo Torresani +98
We present Ego-Exo4D, a diverse, large-scale multimodal multiview video dataset and benchmark challenge. Ego-Exo4D centers around simultaneously-captured egocentric and exocentric…
Universal Humanoid Motion Representations for Physics-Based Control
Zhengyi Luo, Jinkun Cao, Josh Merel +4
We present a universal motion representation that encompasses a comprehensive range of motor skills for physics-based humanoid control. Due to the high dimensionality of humanoids…
Perpetual Humanoid Control for Real-time Simulated Avatars
Zhengyi Luo, Jinkun Cao, Alexander Winkler +2
We present a physics-based humanoid controller that achieves high-fidelity motion imitation and fault-tolerant behavior in the presence of noisy input (e.g. pose estimates from vid…
Trace and Pace: Controllable Pedestrian Animation via Guided Trajectory Diffusion
Davis Rempe, Zhengyi Luo, Xue Bin Peng +5
We introduce a method for generating realistic pedestrian trajectories and full-body animations that can be controlled to meet user-defined goals. We draw on recent advances in gui…
From Universal Humanoid Control to Automatic Physically Valid Character Creation
Zhengyi Luo, Ye Yuan, Kris M. Kitani
Automatically designing virtual humans and humanoids holds great potential in aiding the character creation process in games, movies, and robots. In some cases, a character creator…
Embodied Scene-aware Human Pose Estimation
Zhengyi Luo, Shun Iwase, Ye Yuan +1
We propose embodied scene-aware human pose estimation where we estimate 3D poses based on a simulated agent's proprioception and scene awareness, along with external third-person o…
Transform2Act: Learning a Transform-and-Control Policy for Efficient Agent Design
Ye Yuan, Yuda Song, Zhengyi Luo +2
An agent's functionality is largely determined by its design, i.e., skeletal structure and joint attributes (e.g., length, size, strength). However, finding the optimal agent desig…
Dynamics-Regulated Kinematic Policy for Egocentric Pose Estimation
Zhengyi Luo, Ryo Hachiuma, Ye Yuan +1
We propose a method for object-aware 3D egocentric pose estimation that tightly integrates kinematics modeling, dynamics modeling, and scene object information. Unlike prior kinema…
Kinematics-Guided Reinforcement Learning for Object-Aware 3D Ego-Pose Estimation
Zhengyi Luo, Ryo Hachiuma, Ye Yuan +2
We propose a method for incorporating object interaction and human body dynamics into the task of 3D ego-pose estimation using a head-mounted camera. We use a kinematics model of t…
3D Human Motion Estimation via Motion Compression and Refinement
Zhengyi Luo, S. Alireza Golestaneh, Kris M. Kitani
We develop a technique for generating smooth and accurate 3D human pose and motion estimates from RGB video sequences. Our method, which we call Motion Estimation via Variational A…
Learning Shape Representations for Clothing Variations in Person Re-Identification
Yu-Jhe Li, Zhengyi Luo, Xinshuo Weng +1
Person re-identification (re-ID) aims to recognize instances of the same person contained in multiple images taken across different cameras. Existing methods for re-ID tend to rely…
Cross-Domain 3D Equivariant Image Embeddings
Carlos Esteves, Avneesh Sud, Zhengyi Luo +2
Spherical convolutional networks have been introduced recently as tools to learn powerful feature representations of 3D shapes. Spherical CNNs are equivariant to 3D rotations makin…
Cloud Chaser: Real Time Deep Learning Computer Vision on Low Computing Power Devices
Zhengyi Luo, Austin Small, Liam Dugan +1
Internet of Things(IoT) devices, mobile phones, and robotic systems are often denied the power of deep learning algorithms due to their limited computing power. However, to provide…