GSRFormer: Grounded Situation Recognition Transformer with Alternate Semantic Attention Refinement

Abstract

Grounded Situation Recognition (GSR) aims to generate structured semantic summaries of images for "human-like" event understanding. Specifically, GSR task not only detects the salient activity verb (e.g. buying), but also predicts all corresponding semantic roles (e.g. agent and goods). Inspired by object detection and image captioning tasks, existing methods typically employ a two-stage framework: 1) detect the activity verb, and then 2) predict semantic roles based on the detected verb. Obviously, this illogical framework constitutes a huge obstacle to semantic understanding. First, pre-detecting verbs solely without semantic roles inevitably fails to distinguish many similar daily activities (e.g., offering and giving, buying and selling). Second, predicting semantic roles in a closed auto-regressive manner can hardly exploit the semantic relations among the verb and roles. To this end, in this paper we propose a novel two-stage framework that focuses on utilizing such bidirectional relations within verbs and roles. In the first stage, instead of pre-detecting the verb, we postpone the detection step and assume a pseudo label, where an intermediate representation for each corresponding semantic role is learned from images. In the second stage, we exploit transformer layers to unearth the potential semantic relations within both verbs and semantic roles. With the help of a set of support images, an alternate learning scheme is designed to simultaneously optimize the results: update the verb using nouns corresponding to the image, and update nouns using verbs from support images. Extensive experimental results on challenging SWiG benchmarks show that our renovated framework outperforms other state-of-the-art methods under various metrics.

arXiv commentACM Multimedia 2022 (Oral), Code: https://github.com/zhiqic/GSRFormer

References in corpus (8)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Neural Message Passing for Quantum Chemistry
• Deformable DETR: Deformable Transformers for End-to-End Object Detection
• Temporal Pyramid Network for Action Recognition
• Video as Conditional Graph Hierarchy for Multi-Granular Question Answering
• GNAS: A Greedy Neural Architecture Search Method for Multi-Attribute Learning
• From Show to Tell: A Survey on Deep Learning-based Image Captioning
• Grounded Situation Recognition with Transformers

Cited by in corpus (2)

• Causal Debiasing for Visual Commonsense Reasoning
• Joint Learning for Scattered Point Cloud Understanding with Hierarchical Self-Distillation