92 citations · 365 across the 53 of their papers we have counts for
27 papers · 1 filter
Which way is `right'?: Uncovering limitations of Vision-and-Language Navigation model
Meera Hahn, Amit Raj, James M. Rehg
The challenging task of Vision-and-Language Navigation (VLN) requires embodied agents to follow natural language instructions to reach a goal location or object (e.g. `walk down th…
Listen to Look into the Future: Audio-Visual Egocentric Gaze Anticipation
Bolin Lai, Fiona Ryan, Wenqi Jia +2
Egocentric gaze anticipation serves as a key building block for the emerging capability of Augmented Reality. Notably, gaze behavior is driven by both visual cues and audio signals…
ShapeClipper: Scalable 3D Shape Learning from Single-View Images via Geometric and CLIP-based Consistency
Zixuan Huang, Varun Jampani, Anh Thai +3
We present ShapeClipper, a novel method that reconstructs 3D object shapes from real-world single-view RGB images. Instead of relying on laborious 3D, multi-view or camera pose ann…
Egocentric Auditory Attention Localization in Conversations
Fiona Ryan, Hao Jiang, Abhinav Shukla +2
In a noisy conversation environment such as a dinner party, people often exhibit selective auditory attention, or the ability to focus on a particular speaker while tuning out othe…
Learning Dense Object Descriptors from Multiple Views for Low-shot Category Generalization
Stefan Stojanov, Anh Thai, Zixuan Huang +1
A hallmark of the deep learning era for computer vision is the successful use of large-scale labeled datasets to train feature representations for tasks ranging from object recogni…
Transformer-based Localization from Embodied Dialog with Large-scale Pre-training
Meera Hahn, James M. Rehg
We address the challenging task of Localization via Embodied Dialog (LED). Given a dialog from two agents, an Observer navigating through an unknown environment and a Locator who i…
In the Eye of Transformer: Global-Local Correlation for Egocentric Gaze Estimation
Bolin Lai, Miao Liu, Fiona Ryan +1
In this paper, we present the first transformer-based model to address the challenging problem of egocentric gaze estimation. We observe that the connection between the global scen…
Planes vs. Chairs: Category-guided 3D shape learning without any 3D cues
Zixuan Huang, Stefan Stojanov, Anh Thai +2
We present a novel 3D shape reconstruction method which learns to predict an implicit 3D shape representation from a single RGB image. Our approach uses a set of single-view images…
Ego4D: Around the World in 3,000 Hours of Egocentric Video
Kristen Grauman, Andrew Westbury, Eugene Byrne +82
We introduce Ego4D, a massive-scale egocentric video dataset and benchmark suite. It offers 3,670 hours of daily-life activity video spanning hundreds of scenarios (household, outd…
Using Shape to Categorize: Low-Shot Learning with an Explicit Shape Bias
Stefan Stojanov, Anh Thai, James M. Rehg
It is widely accepted that reasoning about object shape is important for object recognition. However, the most powerful object recognition methods today do not explicitly make use…
4D Human Body Capture from Egocentric Video via 3D Scene Grounding
Miao Liu, Dexin Yang, Yan Zhang +3
We introduce a novel task of reconstructing a time series of second-person 3D human body meshes from monocular egocentric videos. The unique viewpoint and rapid embodied camera mot…
Where Are You? Localization from Embodied Dialog
Meera Hahn, Jacob Krantz, Dhruv Batra +4
We present Where Are You? (WAY), a dataset of ~6k dialogs in which two humans -- an Observer and a Locator -- complete a cooperative localization task. The Observer is spawned at r…
Forecasting Human-Object Interaction: Joint Prediction of Motor Attention and Actions in First Person Video
Miao Liu, Siyu Tang, Yin Li +1
We address the challenging task of anticipating human-object interaction in first person videos. Most existing methods ignore how the camera wearer interacts with the objects, or s…
Unsupervised 3D Pose Estimation with Geometric Self-Supervision
Ching-Hang Chen, Ambrish Tyagi, Amit Agrawal +4
We present an unsupervised learning approach to recover 3D human pose from 2D skeletal joints extracted from a single image. Our method does not require any multi-view image data,…
Tripping through time: Efficient Localization of Activities in Videos
Meera Hahn, Asim Kadav, James M. Rehg +1
Localizing moments in untrimmed videos via language queries is a new and interesting task that requires the ability to accurately ground language into video. Previous works have ap…
Learning to Generate Synthetic Data via Compositing
Shashank Tripathi, Siddhartha Chandra, Amit Agrawal +3
We present a task-aware approach to synthetic data generation. Our framework employs a trainable synthesizer network that is optimized to produce meaningful training samples by ass…
Attention Distillation for Learning Video Representations
Miao Liu, Xin Chen, Yun Zhang +2
We address the challenging problem of learning motion representations using deep models for video recognition. To this end, we make use of attention modules that learn to highlight…
Action2Vec: A Crossmodal Embedding Approach to Action Learning
Meera Hahn, Andrew Silva, James M. Rehg
We describe a novel cross-modal embedding space for actions, named Action2Vec, which combines linguistic cues from class labels with spatio-temporal features derived from video cli…
Taking a Deeper Look at the Inverse Compositional Algorithm
Zhaoyang Lv, Frank Dellaert, James M. Rehg +1
In this paper, we provide a modern synthesis of the classic inverse compositional algorithm for dense image alignment. We first discuss the assumptions made by this well-establishe…
Learning to Localize and Align Fine-Grained Actions to Sparse Instructions
Meera Hahn, Nataniel Ruiz, Jean-Baptiste Alayrac +2
Automatic generation of textual video descriptions that are time-aligned with video content is a long-standing goal in computer vision. The task is challenging due to the difficult…
Connecting Gaze, Scene, and Attention: Generalized Attention Estimation via Joint Modeling of Gaze and Scene Saliency
Eunji Chong, Nataniel Ruiz, Yongxin Wang +3
This paper addresses the challenging problem of estimating the general visual attention of people in images. Our proposed method is designed to work across multiple naturalistic so…
Decoupled Networks
Weiyang Liu, Zhen Liu, Zhiding Yu +5
Inner product-based convolution has been a central component of convolutional neural networks (CNNs) and the key to learning visual representations. Inspired by the observation tha…
Learning Rigidity in Dynamic Scenes with a Moving Camera for 3D Motion Field Estimation
Zhaoyang Lv, Kihwan Kim, Alejandro Troccoli +3
Estimation of 3D motion in a dynamic scene from a temporal pair of images is a core task in many scene understanding problems. In real world applications, a dynamic scene is common…
Fine-Grained Head Pose Estimation Without Keypoints
Nataniel Ruiz, Eunji Chong, James M. Rehg
Estimating the head pose of a person is a crucial problem that has a large amount of applications such as aiding in gaze estimation, modeling attention, fitting 3D models to video…