Visual Semantic Role Labeling

Abstract

In this paper we introduce the problem of Visual Semantic Role Labeling: given an image we want to detect people doing actions and localize the objects of interaction. Classical approaches to action recognition either study the task of action classification at the image or video clip level or at best produce a bounding box around the person doing the action. We believe such an output is inadequate and a complete understanding can only come when we are able to associate objects in the scene to the different semantic roles of the action. To enable progress towards this goal, we annotate a dataset of 16K people instances in 10K images with actions they are doing and associate objects in the scene with different semantic roles for each action. Finally, we provide a set of baseline algorithms for this task and analyze error modes providing directions for future work.

Cited by in corpus (64)

• The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scale
• Transferable Interactiveness Knowledge for Human-Object Interaction Detection
• HOI Analysis: Integrating and Decomposing Human-Object Interaction
• Detecting and Recognizing Human-Object Interactions
• Learning Human-Object Interactions by Graph Parsing Neural Networks
• CholecTriplet2022: Show me a tool and tell me the triplet -- an endoscopic vision challenge for surgical action triplet detection
• HAKE: Human Activity Knowledge Engine
• Learning Human-Object Interaction Detection using Interaction Points
• Mining the Benefits of Two-stage and One-stage HOI Detection
• PPDM: Parallel Point Detection and Matching for Real-time Human-Object Interaction Detection
• ST-HOI: A Spatial-Temporal Baseline for Human-Object Interaction Detection in Videos
• Learning to Detect Human-Object Interactions
• Deep Contextual Attention for Human-Object Interaction Detection
• Situation Recognition with Graph Neural Networks
• HOTR: End-to-End Human-Object Interaction Detection with Transformers
• Pose-aware Multi-level Feature Network for Human Object Interaction Detection
• DRG: Dual Relation Graph for Human-Object Interaction Detection
• Care about you: towards large-scale human-centric visual relationship detection
• QPIC: Query-Based Pairwise Human-Object Interaction Detection with Image-Wide Contextual Information
• Detailed 2D-3D Joint Representation for Human-Object Interaction
• Human-Object Interaction Prediction in Videos through Gaze Following
• Interact as You Intend: Intention-Driven Human-Object Interaction Detection
• VSGNet: Spatial Attention Network for Detecting Human Object Interactions Using Graph Convolutions
• Detecting Human-Object Interactions with Action Co-occurrence Priors
• Actor-Centric Relation Network
• Visual Compositional Learning for Human-Object Interaction Detection
• Reformulating HOI Detection as Adaptive Set Prediction
• PaStaNet: Toward Human Activity Knowledge Engine
• Asynchronous Temporal Fields for Action Recognition
• Pose-based Modular Network for Human-Object Interaction Detection
• Cascaded Human-Object Interaction Recognition
• Contextual Heterogeneous Graph Network for Human-Object Interaction Detection
• Affordance Transfer Learning for Human-Object Interaction Detection
• Understanding Human Hands in Contact at Internet Scale
• Effective Actor-centric Human-object Interaction Detection
• A Local-to-Global Approach to Multi-modal Movie Scene Segmentation
• Detecting Human-Object Interaction via Fabricated Compositional Learning
• Glance and Gaze: Inferring Action-aware Points for One-Stage Human-Object Interaction Detection
• Active Object Localization in Visual Situations
• Commonly Uncommon: Semantic Sparsity in Situation Recognition
• Towards Overcoming False Positives in Visual Relationship Detection
• End-to-End Human Object Interaction Detection with HOI Transformer
• Human-like Controllable Image Captioning with Verb-specific Semantic Roles
• ACP++: Action Co-occurrence Priors for Human-Object Interaction Detection
• Novel Human-Object Interaction Detection via Adversarial Domain Generalization
• Inferring spatial relations from textual descriptions of images
• Probing Image-Language Transformers for Verb Understanding
• GTNet:Guided Transformer Network for Detecting Human-Object Interactions
• Compensating Supervision Incompleteness with Prior Knowledge in Semantic Image Interpretation
• A Structured Model For Action Detection
• Tell Me What They're Holding: Weakly-supervised Object Detection with Transferable Knowledge from Human-object Interaction
• Turbo Learning Framework for Human-Object Interactions Recognition and Human Pose Estimation
• Efficient Human-Object-Interaction (EHOI) Detection via Interaction Label Coding and Conditional Decision
• Spatial Priming for Detecting Human-Object Interactions
• Detecting Human-to-Human-or-Object (H2O) Interactions with DIABOLO
• Human-centric Relation Segmentation: Dataset and Solution
• Annotation Methodologies for Vision and Language Dataset Creation
• Image Captioning with Integrated Bottom-Up and Multi-level Residual Top-Down Attention for Game Scene Understanding
• Human Object Interaction Detection using Two-Direction Spatial Enhancement and Exclusive Object Prior
• Human-Object Interaction Detection via Weak Supervision
• Weakly Supervised Visual Semantic Parsing
• Virtual Multi-Modality Self-Supervised Foreground Matting for Human-Object Interaction
• Visual Understanding and Narration: A Deeper Understanding and Explanation of Visual Scenes
• Loss Guided Activation for Action Recognition in Still Images