EfficientCLIP: Efficient Cross-Modal Pre-training by Ensemble Confident Learning and Language Modeling
arXiv:2109.04699
Abstract
While large scale pre-training has achieved great achievements in bridging the gap between vision and language, it still faces several challenges. First, the cost for pre-training is expensive. Second, there is no efficient way to handle the data noise which degrades model performance. Third, previous methods only leverage limited image-text paired data, while ignoring richer single-modal data, which may result in poor generalization to single-modal downstream tasks. In this work, we propose an EfficientCLIP method via Ensemble Confident Learning to obtain a less noisy data subset. Extra rich non-paired single-modal text data is used for boosting the generalization of text branch. We achieve the state-of-the-art performance on Chinese cross-modal retrieval tasks with only 1/10 training resources compared to CLIP and WenLan, while showing excellent generalization to single-modal tasks, including text retrieval and text classification.
References in corpus (12)
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Learning Transferable Visual Models From Natural Language Supervision
- Language Models are Few-Shot Learners
- Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
- Contrastive Representation Learning: A Framework and Review
- Image Classification with Deep Learning in the Presence of Noisy Labels: A Survey
- COCO-Text: Dataset and Benchmark for Text Detection and Recognition in Natural Images
- How Much Can CLIP Benefit Vision-and-Language Tasks?
- AI Challenger : A Large-scale Dataset for Going Deeper in Image Understanding
- Multimodal Few-Shot Learning with Frozen Language Models
- WenLan: Bridging Vision and Language by Large-Scale Multi-Modal Pre-Training
- Poisoning and Backdooring Contrastive Learning