Dual ResGCN for Balanced Scene GraphGeneration
arXiv:2011.04234
Abstract
Visual scene graph generation is a challenging task. Previous works have achieved great progress, but most of them do not explicitly consider the class imbalance issue in scene graph generation. Models learned without considering the class imbalance tend to predict the majority classes, which leads to a good performance on trivial frequent predicates, but poor performance on informative infrequent predicates. However, predicates of minority classes often carry more semantic and precise information~(\textit{e.g.}, \emph{`on'} v.s \emph{`parked on'}). % which leads to a good score of recall, but a poor score of mean recall. To alleviate the influence of the class imbalance, we propose a novel model, dubbed \textit{dual ResGCN}, which consists of an object residual graph convolutional network and a relation residual graph convolutional network. The two networks are complementary to each other. The former captures object-level context information, \textit{i.e.,} the connections among objects. We propose a novel ResGCN that enhances object features in a cross attention manner. Besides, we stack multiple contextual coefficients to alleviate the imbalance issue and enrich the prediction diversity. The latter is carefully designed to explicitly capture relation-level context information \textit{i.e.,} the connections among relations. We propose to incorporate the prior about the co-occurrence of relation pairs into the graph to further help alleviate the class imbalance issue. Extensive evaluations of three tasks are performed on the large-scale database VG to demonstrate the superiority of the proposed method.
References in corpus (16)
- Deep Residual Learning for Image Recognition
- Dynamic Memory Networks for Visual and Textual Question Answering
- Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learning
- GCNet: Non-local Networks Meet Squeeze-Excitation Networks and Beyond
- Learning to Count Objects in Natural Images for Visual Question Answering
- Libra R-CNN: Towards Balanced Learning for Object Detection
- Visual Relationship Detection with Language Priors
- Scene Graph Generation by Iterative Message Passing
- High-performance Semantic Segmentation Using Very Deep Fully Convolutional Networks
- Visual Translation Embedding Network for Visual Relation Detection
- Image Generation from Scene Graphs
- Knowledge-Embedded Routing Network for Scene Graph Generation
- Unbiased Scene Graph Generation from Biased Training
- Scene Graph Generation with External Knowledge and Image Reconstruction
- Deep Contextual Attention for Human-Object Interaction Detection
- Unpaired Image Captioning by Language Pivoting