Is Object Detection Necessary for Human-Object Interaction Recognition?
arXiv:2107.13083
Abstract
This paper revisits human-object interaction (HOI) recognition at image level without using supervisions of object location and human pose. We name it detection-free HOI recognition, in contrast to the existing detection-supervised approaches which rely on object and keypoint detections to achieve state of the art. With our method, not only the detection supervision is evitable, but superior performance can be achieved by properly using image-text pre-training (such as CLIP) and the proposed Log-Sum-Exp Sign (LSE-Sign) loss function. Specifically, using text embeddings of class labels to initialize the linear classifier is essential for leveraging the CLIP pre-trained image encoder. In addition, LSE-Sign loss facilitates learning from multiple labels on an imbalanced dataset by normalizing gradients over all classes in a softmax format. Surprisingly, our detection-free solution achieves 60.5 mAP on the HICO dataset, outperforming the detection-supervised state of the art by 13.4 mAP
References in corpus (18)
- Adam: A Method for Stochastic Optimization
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Learning Transferable Visual Models From Natural Language Supervision
- Focal Loss for Dense Object Detection
- ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision
- Attentional Pooling for Action Recognition
- Scene Graph Generation by Iterative Message Passing
- Contextual Action Recognition with R*CNN
- Knowledge-Embedded Routing Network for Scene Graph Generation
- Unbiased Scene Graph Generation from Biased Training
- HAKE: Human Activity Knowledge Engine
- PPDM: Parallel Point Detection and Matching for Real-time Human-Object Interaction Detection
- DRG: Dual Relation Graph for Human-Object Interaction Detection
- VIVO: Visual Vocabulary Pre-Training for Novel Object Captioning
- PaStaNet: Toward Human Activity Knowledge Engine
- Reformulating HOI Detection as Adaptive Set Prediction
- Learning Visual Commonsense for Robust Scene Graph Generation
- Pairwise Body-Part Attention for Recognizing Human-Object Interactions