Improving Weakly Supervised Visual Grounding by Contrastive Knowledge Distillation

Abstract

Weakly supervised phrase grounding aims at learning region-phrase correspondences using only image-sentence pairs. A major challenge thus lies in the missing links between image regions and sentence phrases during training. To address this challenge, we leverage a generic object detector at training time, and propose a contrastive learning framework that accounts for both region-phrase and image-sentence matching. Our core innovation is the learning of a region-phrase score function, based on which an image-sentence score function is further constructed. Importantly, our region-phrase score function is learned by distilling from soft matching scores between the detected object names and candidate phrases within an image-sentence pair, while the image-sentence score function is supervised by ground-truth image-sentence pairs. The design of such score functions removes the need of object detection at test time, thereby significantly reducing the inference cost. Without bells and whistles, our approach achieves state-of-the-art results on visual phrase grounding, surpassing previous methods that require expensive object detectors at test time.

References in corpus (16)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Distilling the Knowledge in a Neural Network
• Representation Learning with Contrastive Predictive Coding
• FitNets: Hints for Thin Deep Nets
• NLTK: The Natural Language Toolkit
• Learning deep representations by mutual information estimation and maximization
• Microsoft COCO Captions: Data Collection and Evaluation Server
• Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer
• Do Deep Nets Really Need to be Deep?
• Deep Fragment Embeddings for Bidirectional Image Sentence Mapping
• Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations
• Weakly-Supervised Video Object Grounding from Text by Loss Weighting and Object Interaction
• Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models
• Weakly-supervised Visual Grounding of Phrases with Linguistic Structures
• Large-Scale Domain Adaptation via Teacher-Student Learning
• Improving One-stage Visual Grounding by Recursive Sub-query Construction

Cited by in corpus (1)

• Relation-aware Instance Refinement for Weakly Supervised Visual Grounding