Visually Grounded Neural Syntax Acquisition

Abstract

We present the Visually Grounded Neural Syntax Learner (VG-NSL), an approach for learning syntactic representations and structures without any explicit supervision. The model learns by looking at natural images and reading paired captions. VG-NSL generates constituency parse trees of texts, recursively composes representations for constituents, and matches them with images. We define concreteness of constituents by their matching scores with images, and use it to guide the parsing of text. Experiments on the MSCOCO data set show that VG-NSL outperforms various unsupervised parsing approaches that do not use visual grounding, in terms of F1 scores against gold parse trees. We find that VGNSL is much more stable with respect to the choice of random initialization and the amount of training data. We also find that the concreteness acquired by VG-NSL correlates well with a similar measure defined by linguists. Finally, we also apply VG-NSL to multiple languages in the Multi30K data set, showing that our model consistently outperforms prior unsupervised approaches.

arXiv commentACL 2019. Project page: https://ttic.uchicago.edu/~freda/project/vgnsl/

References in corpus (8)

• Adam: A Method for Stochastic Optimization
• Categorical Reparameterization with Gumbel-Softmax
• Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
• VSE++: Improving Visual-Semantic Embeddings with Hard Negatives
• Ask Your Neurons: A Neural-based Approach to Answering Questions about Images
• Learning Visually-Grounded Semantics from Contrastive Adversarial Samples
• Resolving Language and Vision Ambiguities Together: Joint Segmentation & Prepositional Attachment Resolution in Captioned Scenes
• Word learning and the acquisition of syntactic--semantic overhypotheses

Cited by in corpus (22)

• Universal Multimodal Representation for Language Understanding
• Are Pre-trained Language Models Aware of Phrases? Simple but Strong Baselines for Grammar Induction
• A Critical Analysis of Biased Parsers in Unsupervised Parsing
• Vokenization: Improving Language Understanding with Contextualized, Visual-Grounded Supervision
• Compound Probabilistic Context-Free Grammars for Grammar Induction
• Analogical Reasoning for Visually Grounded Language Acquisition
• KANDINSKYPatterns -- An experimental exploration environment for Pattern Analysis and Machine Intelligence
• Visually Grounded Compound PCFGs
• Probing Contextualized Sentence Representations with Visual Awareness
• Grammar-Based Grounded Lexicon Learning
• Tree-Augmented Cross-Modal Encoding for Complex-Query Video Retrieval
• On the Role of Supervision in Unsupervised Constituency Parsing
• The Importance of Category Labels in Grammar Induction with Child-directed Utterances
• What is Learned in Visually Grounded Neural Syntax Acquisition
• Neural Bi-Lexicalized PCFG Induction
• Learning Syntax from Naturally-Occurring Bracketings
• Improved Latent Tree Induction with Distant Supervision via Span Constraints
• Dependency Induction Through the Lens of Visual Perception
• Grounding 'Grounding' in NLP
• Data-efficient Alignment of Multimodal Sequences by Aligning Gradient Updates and Internal Feature Distributions
• Sequence-to-Sequence Learning with Latent Neural Grammars
• Video-aided Unsupervised Grammar Induction