POV: Prompt-Oriented View-Agnostic Learning for Egocentric Hand-Object Interaction in the Multi-View World

Abstract

We humans are good at translating third-person observations of hand-object interactions (HOI) into an egocentric view. However, current methods struggle to replicate this ability of view adaptation from third-person to first-person. Although some approaches attempt to learn view-agnostic representation from large-scale video datasets, they ignore the relationships among multiple third-person views. To this end, we propose a Prompt-Oriented View-agnostic learning (POV) framework in this paper, which enables this view adaptation with few egocentric videos. Specifically, We introduce interactive masking prompts at the frame level to capture fine-grained action information, and view-aware prompts at the token level to learn view-agnostic representation. To verify our method, we establish two benchmarks for transferring from multiple third-person views to the egocentric view. Our extensive experiments on these benchmarks demonstrate the efficiency and effectiveness of our POV framework and prompt tuning techniques in terms of view adaptation and view generalization. Our code is available at \url{https://github.com/xuboshen/pov_acmmm2023}.

arXiv commentAccepted by ACM MM 2023. Project page: https://xuboshen.github.io/

References in corpus (5)

• NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding
• HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips
• Rolling-Unrolling LSTMs for Action Anticipation from First-Person Video
• Egocentric Video-Language Pretraining
• Learning Viewpoint-Agnostic Visual Representations by Recovering Tokens in 3D Space

Cited by in corpus (1)

• Cognition Transferring and Decoupling for Text-supervised Egocentric Semantic Segmentation