Transforming Visual Scene Graphs to Image Captions
arXiv:2305.02177 · doi:10.18653/v1/2023.acl-long.694
Abstract
We propose to Transform Scene Graphs (TSG) into more descriptive captions. In TSG, we apply multi-head attention (MHA) to design the Graph Neural Network (GNN) for embedding scene graphs. After embedding, different graph embeddings contain diverse specific knowledge for generating the words with different part-of-speech, e.g., object/attribute embedding is good for generating nouns/adjectives. Motivated by this, we design a Mixture-of-Expert (MOE)-based decoder, where each expert is built on MHA, for discriminating the graph embeddings to generate different kinds of words. Since both the encoder and decoder are built based on the MHA, as a result, we construct a homogeneous encoder-decoder unlike the previous heterogeneous ones which usually apply Fully-Connected-based GNN and LSTM-based decoder. The homogeneous architecture enables us to unify the training configuration of the whole model instead of specifying different training strategies for diverse sub-networks as in the heterogeneous pipeline, which releases the training difficulty. Extensive experiments on the MS-COCO captioning benchmark validate the effectiveness of our TSG. The code is in: https://github.com/GaryJiajia/TSG.
arXiv comment12 pages, 4 figures, has been accepted by ACL 2023 main conference
References in corpus (8)
- Adam: A Method for Stochastic Optimization
- Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
- Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
- GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
- GLaM: Efficient Scaling of Language Models with Mixture-of-Experts
- mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality
- Show, Recall, and Tell: Image Captioning with Recall Mechanism
- mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and Video