Relation Transformer Network
arXiv:2004.06193
Abstract
The extraction of a scene graph with objects as nodes and mutual relationships as edges is the basis for a deep understanding of image content. Despite recent advances, such as message passing and joint classification, the detection of visual relationships remains a challenging task due to sub-optimal exploration of the mutual interaction among the visual objects. In this work, we propose a novel transformer formulation for scene graph generation and relation prediction. We leverage the encoder-decoder architecture of the transformer for rich feature embedding of nodes and edges. Specifically, we model the node-to-node interaction with the self-attention of the transformer encoder and the edge-to-node interaction with the cross-attention of the transformer decoder. Further, we introduce a novel positional embedding suitable to handle edges in the decoder. Finally, our relation prediction module classifies the directed relation from the learned node and edge embedding. We name this architecture as Relation Transformer Network (RTN). On the Visual Genome and GQA dataset, we have achieved an overall mean of 4.85% and 3.1% point improvement in comparison with state-of-the-art methods. Our experiments show that Relation Transformer can efficiently model context across various datasets with small, medium, and large-scale relation classification.
References in corpus (12)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- Semi-Supervised Classification with Graph Convolutional Networks
- Empirical Evaluation of Rectified Activations in Convolutional Network
- Pixels to Graphs by Associative Embedding
- ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text Data
- Mapping Images to Scene Graphs with Permutation-Invariant Structured Prediction
- Graph Density-Aware Losses for Novel Compositions in Scene Graph Generation
- Generating Natural Language Explanations for Visual Question Answering using Scene Graphs and Visual Attention
- Scene Graph Reasoning for Visual Question Answering
- Scenes and Surroundings: Scene Graph Generation using Relation Transformer