GTNet:Guided Transformer Network for Detecting Human-Object Interactions

Abstract

The human-object interaction (HOI) detection task refers to localizing humans, localizing objects, and predicting the interactions between each human-object pair. HOI is considered one of the fundamental steps in truly understanding complex visual scenes. For detecting HOI, it is important to utilize relative spatial configurations and object semantics to find salient spatial regions of images that highlight the interactions between human object pairs. This issue is addressed by the novel self-attention based guided transformer network, GTNet. GTNet encodes this spatial contextual information in human and object visual features via self-attention while achieving state of the art results on both the V-COCO and HICO-DET datasets. Code will be made available online.

arXiv commentaccepted for presentation in Pattern Recognition and Tracking XXXIV at SPIE commerce+ defence Program

References in corpus (12)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
• Visual Semantic Role Labeling
• Training Tips for the Transformer Model
• iCAN: Instance-Centric Attention Network for Human-Object Interaction Detection
• Attentional Pooling for Action Recognition
• Learning Deep Transformer Models for Machine Translation
• Collaborative Spatio-temporal Feature Learning for Video Action Recognition
• Visual Compositional Learning for Human-Object Interaction Detection
• DIRV: Dense Interaction Region Voting for End-to-End Human-Object Interaction Detection