Tackling the Challenges in Scene Graph Generation with Local-to-Global Interactions
arXiv:2106.08543 · doi:10.1109/TNNLS.2022.3159990
Abstract
In this work, we seek new insights into the underlying challenges of the Scene Graph Generation (SGG) task. Quantitative and qualitative analysis of the Visual Genome dataset implies -- 1) Ambiguity: even if inter-object relationship contains the same object (or predicate), they may not be visually or semantically similar, 2) Asymmetry: despite the nature of the relationship that embodied the direction, it was not well addressed in previous studies, and 3) Higher-order contexts: leveraging the identities of certain graph elements can help to generate accurate scene graphs. Motivated by the analysis, we design a novel SGG framework, Local-to-Global Interaction Networks (LOGIN). Locally, interactions extract the essence between three instances of subject, object, and background, while baking direction awareness into the network by explicitly constraining the input order of subject and object. Globally, interactions encode the contexts between every graph component (i.e., nodes and edges). Finally, Attract & Repel loss is utilized to fine-tune the distribution of predicate embeddings. By design, our framework enables predicting the scene graph in a bottom-up manner, leveraging the possible complementariness. To quantify how much LOGIN is aware of relational direction, a new diagnostic task called Bidirectional Relationship Classification (BRC) is also proposed. Experimental results demonstrate that LOGIN can successfully distinguish relational direction than existing methods (in BRC task), while showing state-of-the-art results on the Visual Genome benchmark (in SGG task).
arXiv commentIEEE Transactions on Neural Networks and Learning Systems (TNNLS)
References in corpus (22)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
- Neural Message Passing for Quantum Chemistry
- Fully Convolutional Networks for Semantic Segmentation
- A simple neural network module for relational reasoning
- Visual Relationship Detection with Language Priors
- DeepGCNs: Can GCNs Go as Deep as CNNs?
- Scene Graph Generation by Iterative Message Passing
- Detecting Visual Relationships with Deep Relational Networks
- Deep Set Prediction Networks
- Visual Translation Embedding Network for Visual Relation Detection
- Deep Variation-structured Reinforcement Learning for Visual Relationship and Attribute Detection
- Image Generation from Scene Graphs
- Knowledge-Embedded Routing Network for Scene Graph Generation
- Factorizable Net: An Efficient Subgraph-based Framework for Scene Graph Generation
- Unbiased Scene Graph Generation from Biased Training
- Looking for the Devil in the Details: Learning Trilinear Attention Sampling Network for Fine-grained Image Recognition
- Transferable Interactiveness Knowledge for Human-Object Interaction Detection
- PPR-FCN: Weakly Supervised Visual Relation Detection via Parallel Pairwise R-FCN
- Bridging Knowledge Graphs to Generate Scene Graphs
- Sketching Image Gist: Human-Mimetic Hierarchical Scene Graph Generation
- Context-Aware Group Captioning via Self-Attention and Contrastive Features