5 citations · 20 across the 12 of their papers we have counts for
6 papers · 1 filter
Pretrained Language Models as Visual Planners for Human Assistance
Dhruvesh Patel, Hamid Eghbalzadeh, Nitin Kamra +3
In our pursuit of advancing multi-modal AI assistants capable of guiding users to achieve complex multi-step goals, we propose the task of "Visual Planning for Assistance (VPA)". G…
Action Dynamics Task Graphs for Learning Plannable Representations of Procedural Tasks
Weichao Mao, Ruta Desai, Michael Louis Iuzzolino +1
Given video demonstrations and paired narrations of an at-home procedural task such as changing a tire, we present an approach to extract the underlying task structure -- relevant…
Learning a Visually Grounded Memory Assistant
Meera Hahn, Kevin Carlberg, Ruta Desai +1
We introduce a novel interface for large scale collection of human memory and assistance. Using the 3D Matterport simulator we create a realistic indoor environments in which we ha…
EgoEnv: Human-centric environment representations from egocentric video
Tushar Nagarajan, Santhosh Kumar Ramakrishnan, Ruta Desai +2
First-person video highlights a camera-wearer's activities in the context of their persistent environment. However, current video understanding approaches reason over visual featur…
Episodic Memory Question Answering
Samyak Datta, Sameer Dharur, Vincent Cartillier +4
Egocentric augmented reality devices such as wearable glasses passively capture visual data as a human wearer tours a home environment. We envision a scenario wherein the human com…
How You Move Your Head Tells What You Do: Self-supervised Video Representation Learning with Egocentric Cameras and IMU Sensors
Satoshi Tsutsui, Ruta Desai, Karl Ridgeway
Understanding users' activities from head-mounted cameras is a fundamental task for Augmented and Virtual Reality (AR/VR) applications. A typical approach is to train a classifier…