Zero-Shot Scene Graph Generation via Triplet Calibration and Reduction

Abstract

Scene Graph Generation (SGG) plays a pivotal role in downstream vision-language tasks. Existing SGG methods typically suffer from poor compositional generalizations on unseen triplets. They are generally trained on incompletely annotated scene graphs that contain dominant triplets and tend to bias toward these seen triplets during inference. To address this issue, we propose a Triplet Calibration and Reduction (T-CAR) framework in this paper. In our framework, a triplet calibration loss is first presented to regularize the representations of diverse triplets and to simultaneously excavate the unseen triplets in incompletely annotated training scene graphs. Moreover, the unseen space of scene graphs is usually several times larger than the seen space since it contains a huge number of unrealistic compositions. Thus, we propose an unseen space reduction loss to shift the attention of excavation to reasonable unseen compositions to facilitate the model training. Finally, we propose a contextual encoder to improve the compositional generalizations of unseen triplets by explicitly modeling the relative spatial relations between subjects and objects. Extensive experiments show that our approach achieves consistent improvements for zero-shot SGG over state-of-the-art methods. The code is available at https://github.com/jkli1998/T-CAR.

arXiv commentAccept in TOMM 2023

References in corpus (7)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Learning from positive and unlabeled data: a survey
• A Review of Generalized Zero-Shot Learning Methods
• A Comprehensive Survey of Scene Graphs: Generation and Application
• Contextual Translation Embedding for Visual Relationship Detection and Scene Graph Generation
• Graph Density-Aware Losses for Novel Compositions in Scene Graph Generation
• Answer Questions with Right Image Regions: A Visual Attention Regularization Approach