Connecting Language and Vision for Natural Language-Based Vehicle Retrieval

Abstract

Vehicle search is one basic task for the efficient traffic management in terms of the AI City. Most existing practices focus on the image-based vehicle matching, including vehicle re-identification and vehicle tracking. In this paper, we apply one new modality, i.e., the language description, to search the vehicle of interest and explore the potential of this task in the real-world scenario. The natural language-based vehicle search poses one new challenge of fine-grained understanding of both vision and language modalities. To connect language and vision, we propose to jointly train the state-of-the-art vision models with the transformer-based language model in an end-to-end manner. Except for the network structure design and the training strategy, several optimization objectives are also re-visited in this work. The qualitative and quantitative experiments verify the effectiveness of the proposed method. Our proposed method has achieved the 1st place on the 5th AI City Challenge, yielding competitive performance 18.69% MRR accuracy on the private test set. We hope this work can pave the way for the future study on using language description effectively and efficiently for real-world vehicle retrieval systems. The code will be available at https://github.com/ShuaiBai623/AIC2021-T5-CLV.

arXiv commentCVPR 2021 AI CITY CHALLENGE Natural Language-Based Vehicle Retrieval Top 1

References in corpus (19)

• Learning Transferable Visual Models From Natural Language Supervision
• Barlow Twins: Self-Supervised Learning via Redundancy Reduction
• Learning Video Representations using Contrastive Bidirectional Transformer
• Learning a Text-Video Embedding from Incomplete and Heterogeneous Data
• VideoBERT: A Joint Model for Video and Language Representation Learning
• MDMMT: Multidomain Multimodal Transformer for Video Retrieval
• TinyBERT: Distilling BERT for Natural Language Understanding
• The Devil is in the Middle: Exploiting Mid-level Representations for Cross-Domain Instance Matching
• Weakly-Supervised Video Object Grounding from Text by Loss Weighting and Object Interaction
• HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-training
• Less is More: ClipBERT for Video-and-Language Learning via Sparse Sampling
• M6: A Chinese Multimodal Pretrainer
• Learning 2D Temporal Adjacent Networks for Moment Localization with Natural Language
• Text Data Augmentation Made Simple By Leveraging NLP Cloud APIs
• MAN: Moment Alignment Network for Natural Language Moment Retrieval via Iterative Graph Adjustment
• End-to-end Concept Word Detection for Video Captioning, Retrieval, and Question Answering
• Weakly-Supervised Spatio-Temporally Grounding Natural Sentence in Video
• Cross-Modal and Hierarchical Modeling of Video and Text
• Dual Encoding for Zero-Example Video Retrieval