Scene Graph Parsing as Dependency Parsing
arXiv:1803.09189
Abstract
In this paper, we study the problem of parsing structured knowledge graphs from textual descriptions. In particular, we consider the scene graph representation that considers objects together with their attributes and relations: this representation has been proved useful across a variety of vision and language applications. We begin by introducing an alternative but equivalent edge-centric view of scene graphs that connect to dependency parses. Together with a careful redesign of label and action space, we combine the two-stage pipeline used in prior work (generic dependency parsing followed by simple post-processing) into one, enabling end-to-end training. The scene graphs generated by our learned neural dependency parser achieve an F-score similarity of 49.67% to ground truth graphs on our evaluation set, surpassing best previous approaches by 5%. We further demonstrate the effectiveness of our learned parser on image retrieval applications.
arXiv commentTo appear in NAACL 2018 as oral. Code is available at https://github.com/Yusics/bist-parser/tree/sgparser
References in corpus (14)
- Adam: A Method for Stochastic Optimization
- Efficient Estimation of Word Representations in Vector Space
- DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition
- VQA: Visual Question Answering
- Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN)
- Transition-Based Dependency Parsing with Stack Long Short-Term Memory
- One Model To Learn Them All
- Visual Relationship Detection with Language Priors
- Scene Graph Generation by Iterative Message Passing
- A Transition-Based Directed Acyclic Graph Parser for UCCA
- Order-Embeddings of Images and Language
- Attention Correctness in Neural Image Captioning
- Detecting Visual Relationships with Deep Relational Networks
- Fast(er) Exact Decoding and Global Training for Transition-Based Dependency Parsing via a Minimal Feature Set