Learning to Compose and Reason with Language Tree Structures for Visual Grounding

Abstract

Grounding natural language in images, such as localizing "the black dog on the left of the tree", is one of the core problems in artificial intelligence, as it needs to comprehend the fine-grained and compositional language space. However, existing solutions rely on the association between the holistic language features and visual features, while neglect the nature of compositional reasoning implied in the language. In this paper, we propose a natural language grounding model that can automatically compose a binary tree structure for parsing the language and then perform visual reasoning along the tree in a bottom-up fashion. We call our model RVG-TREE: Recursive Grounding Tree, which is inspired by the intuition that any language expression can be recursively decomposed into two constituent parts, and the grounding confidence score can be recursively accumulated by calculating their grounding scores returned by sub-trees. RVG-TREE can be trained end-to-end by using the Straight-Through Gumbel-Softmax estimator that allows the gradients from the continuous score functions passing through the discrete tree construction. Experiments on several benchmarks show that our model achieves the state-of-the-art performance with more explainable reasoning.

Accepted to IEEE Transactions on Pattern Analysis and Machine Intelligence (T-PAMI)

References in corpus (7)

• Adam: A Method for Stochastic Optimization
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Categorical Reparameterization with Gumbel-Softmax
• VQA: Visual Question Answering
• Grounding of Textual Phrases in Images by Reconstruction
• Jointly Learning Sentence Embeddings and Syntax with Unsupervised Tree-LSTMs
• Do latent tree learning models identify meaningful structure in sentences?

Cited by in corpus (21)

• RSVG: Exploring Data and Models for Visual Grounding on Remote Sensing Data
• VLT: Vision-Language Transformer and Query Generation for Referring Segmentation
• 3D-SPS: Single-Stage 3D Visual Grounding via Referred Point Progressive Selection
• CLIP-VG: Self-paced Curriculum Adapting of CLIP for Visual Grounding
• Referring Transformer: A One-step Approach to Multi-task Visual Grounding
• Context Disentangling and Prototype Inheriting for Robust Visual Grounding
• Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods
• Universal Multimodal Representation for Language Understanding
• HiVG: Hierarchical Multimodal Fine-grained Modulation for Visual Grounding
• CityFlow-NL: Tracking and Retrieval of Vehicles at City Scale by Natural Language Descriptions
• Toward Visual Grounding: A Survey
• Visual Grounding with Multi-modal Conditional Adaptation
• Ref-NMS: Breaking Proposal Bottlenecks in Two-Stage Referring Expression Grounding
• Prophet: Prompting Large Language Models with Complementary Answer Heuristics for Knowledge-based Visual Question Answering
• Joint Visual Grounding with Language Scene Graphs
• Graph-Structured Referring Expression Reasoning in The Wild
• Learning to Discretely Compose Reasoning Module Networks for Video Captioning
• Siamese Natural Language Tracker: Tracking by Natural Language Descriptions with Siamese Trackers
• ResVG: Enhancing Relation and Semantic Understanding in Multiple Instances for Visual Grounding
• Discriminative Triad Matching and Reconstruction for Weakly Referring Expression Grounding
• VL-NMS: Breaking Proposal Bottlenecks in Two-Stage Visual-Language Matching